[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260504 19:59:05.213203 11536 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.68.62:40525
I20260504 19:59:05.214947 11536 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 19:59:05.216115 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 19:59:05.236311 11543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:05.236395 11545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:05.236625 11542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:05.236712 11536 server_base.cc:1061] running on GCE node
I20260504 19:59:06.430471 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 19:59:06.430785 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 19:59:06.430953 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924746430935 us; error 0 us; skew 500 ppm
I20260504 19:59:06.431807 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 19:59:06.437644 11536 webserver.cc:492] Webserver started at http://127.11.68.62:35979/ using document root <none> and password file <none>
I20260504 19:59:06.438715 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 19:59:06.438949 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 19:59:06.439503 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 19:59:06.446025 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/master-0-root/instance:
uuid: "42967374b8734f19850eb44b7a55611c"
format_stamp: "Formatted at 2026-05-04 19:59:06 on dist-test-slave-3x8x"
I20260504 19:59:06.454829 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260504 19:59:06.460598 11551 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:06.462167 11536 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260504 19:59:06.462607 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/master-0-root
uuid: "42967374b8734f19850eb44b7a55611c"
format_stamp: "Formatted at 2026-05-04 19:59:06 on dist-test-slave-3x8x"
I20260504 19:59:06.462991 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 19:59:06.519901 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 19:59:06.521631 11536 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 19:59:06.522171 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 19:59:06.622632 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.62:40525
I20260504 19:59:06.622797 11602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.62:40525 every 8 connection(s)
I20260504 19:59:06.636443 11603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:06.673218 11603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c: Bootstrap starting.
I20260504 19:59:06.684017 11603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:06.687549 11603 log.cc:826] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c: Log is configured to *not* fsync() on all Append() calls
I20260504 19:59:06.695744 11603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c: No bootstrap required, opened a new log
I20260504 19:59:06.726157 11603 raft_consensus.cc:359] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42967374b8734f19850eb44b7a55611c" member_type: VOTER }
I20260504 19:59:06.726852 11603 raft_consensus.cc:385] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:06.727133 11603 raft_consensus.cc:740] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42967374b8734f19850eb44b7a55611c, State: Initialized, Role: FOLLOWER
I20260504 19:59:06.728219 11603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42967374b8734f19850eb44b7a55611c" member_type: VOTER }
I20260504 19:59:06.728899 11603 raft_consensus.cc:399] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260504 19:59:06.729208 11603 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260504 19:59:06.729583 11603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:06.736498 11603 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42967374b8734f19850eb44b7a55611c" member_type: VOTER }
I20260504 19:59:06.737519 11603 leader_election.cc:304] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42967374b8734f19850eb44b7a55611c; no voters:
I20260504 19:59:06.741411 11603 leader_election.cc:290] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260504 19:59:06.741926 11606 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:06.749645 11606 raft_consensus.cc:697] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 1 LEADER]: Becoming Leader. State: Replica: 42967374b8734f19850eb44b7a55611c, State: Running, Role: LEADER
I20260504 19:59:06.751679 11606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42967374b8734f19850eb44b7a55611c" member_type: VOTER }
I20260504 19:59:06.753340 11603 sys_catalog.cc:565] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [sys.catalog]: configured and running, proceeding with master startup.
W20260504 19:59:06.778813 11557 tablet.cc:2404] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 19:59:06.777619 11607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42967374b8734f19850eb44b7a55611c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42967374b8734f19850eb44b7a55611c" member_type: VOTER } }
I20260504 19:59:06.776839 11608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42967374b8734f19850eb44b7a55611c. Latest consensus state: current_term: 1 leader_uuid: "42967374b8734f19850eb44b7a55611c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42967374b8734f19850eb44b7a55611c" member_type: VOTER } }
I20260504 19:59:06.781917 11607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [sys.catalog]: This master's current role is: LEADER
I20260504 19:59:06.781917 11608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [sys.catalog]: This master's current role is: LEADER
I20260504 19:59:06.791095 11615 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 19:59:06.815658 11615 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 19:59:06.824119 11536 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260504 19:59:06.850921 11615 catalog_manager.cc:1357] Generated new cluster ID: 8f8d5a88dcd64f80ae6d548b19f1ccd3
I20260504 19:59:06.851394 11615 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 19:59:06.876645 11615 catalog_manager.cc:1380] Generated new certificate authority record
I20260504 19:59:06.879773 11615 catalog_manager.cc:1514] Loading token signing keys...
I20260504 19:59:06.913898 11615 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c: Generated new TSK 0
I20260504 19:59:06.915113 11615 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 19:59:06.962523 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 19:59:06.977257 11625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:06.981032 11626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:06.992051 11628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:06.992286 11536 server_base.cc:1061] running on GCE node
I20260504 19:59:06.993326 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 19:59:06.993578 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 19:59:06.993785 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924746993760 us; error 0 us; skew 500 ppm
I20260504 19:59:06.994451 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 19:59:07.002516 11536 webserver.cc:492] Webserver started at http://127.11.68.1:43443/ using document root <none> and password file <none>
I20260504 19:59:07.003221 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 19:59:07.003491 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 19:59:07.003825 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 19:59:07.005244 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-0-root/instance:
uuid: "f65745628d11455ab10c4c37fc004db4"
format_stamp: "Formatted at 2026-05-04 19:59:06 on dist-test-slave-3x8x"
I20260504 19:59:07.013993 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260504 19:59:07.021643 11633 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:07.022840 11536 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.004s
I20260504 19:59:07.023182 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-0-root
uuid: "f65745628d11455ab10c4c37fc004db4"
format_stamp: "Formatted at 2026-05-04 19:59:06 on dist-test-slave-3x8x"
I20260504 19:59:07.023569 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 19:59:07.061215 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 19:59:07.063438 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 19:59:07.069151 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 19:59:07.077189 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 19:59:07.077442 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:07.077852 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 19:59:07.078058 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260504 19:59:07.313697 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.1:34535
I20260504 19:59:07.315644 11695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.1:34535 every 8 connection(s)
I20260504 19:59:07.328369 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 19:59:07.353183 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:07.357591 11701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:07.365862 11703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:07.369876 11536 server_base.cc:1061] running on GCE node
I20260504 19:59:07.371125 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 19:59:07.371400 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 19:59:07.371623 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924747371596 us; error 0 us; skew 500 ppm
I20260504 19:59:07.372251 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 19:59:07.375993 11536 webserver.cc:492] Webserver started at http://127.11.68.2:35607/ using document root <none> and password file <none>
I20260504 19:59:07.376746 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 19:59:07.377055 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 19:59:07.377374 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 19:59:07.378616 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-1-root/instance:
uuid: "50b6b0a3f3224c01a95509c61e631d12"
format_stamp: "Formatted at 2026-05-04 19:59:07 on dist-test-slave-3x8x"
I20260504 19:59:07.381057 11696 heartbeater.cc:344] Connected to a master server at 127.11.68.62:40525
I20260504 19:59:07.381671 11696 heartbeater.cc:461] Registering TS with master...
I20260504 19:59:07.383420 11696 heartbeater.cc:507] Master 127.11.68.62:40525 requested a full tablet report, sending...
I20260504 19:59:07.386266 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260504 19:59:07.388365 11568 ts_manager.cc:194] Registered new tserver with Master: f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535)
I20260504 19:59:07.392858 11568 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45918
I20260504 19:59:07.392917 11708 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:07.394255 11536 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260504 19:59:07.394666 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-1-root
uuid: "50b6b0a3f3224c01a95509c61e631d12"
format_stamp: "Formatted at 2026-05-04 19:59:07 on dist-test-slave-3x8x"
I20260504 19:59:07.395131 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 19:59:07.434037 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 19:59:07.435861 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 19:59:07.437716 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 19:59:07.440338 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 19:59:07.440587 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:07.440846 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 19:59:07.441030 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:07.683494 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.2:37859
I20260504 19:59:07.683773 11770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.2:37859 every 8 connection(s)
I20260504 19:59:07.701259 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 19:59:07.721726 11774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:07.727959 11775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:07.730036 11771 heartbeater.cc:344] Connected to a master server at 127.11.68.62:40525
I20260504 19:59:07.730618 11771 heartbeater.cc:461] Registering TS with master...
I20260504 19:59:07.732151 11771 heartbeater.cc:507] Master 127.11.68.62:40525 requested a full tablet report, sending...
I20260504 19:59:07.736181 11568 ts_manager.cc:194] Registered new tserver with Master: 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:07.738780 11568 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45920
I20260504 19:59:07.748030 11536 server_base.cc:1061] running on GCE node
W20260504 19:59:07.749037 11777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:07.750670 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 19:59:07.750881 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 19:59:07.751089 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924747751055 us; error 0 us; skew 500 ppm
I20260504 19:59:07.751688 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 19:59:07.754402 11536 webserver.cc:492] Webserver started at http://127.11.68.3:33055/ using document root <none> and password file <none>
I20260504 19:59:07.755003 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 19:59:07.755218 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 19:59:07.755553 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 19:59:07.756713 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-2-root/instance:
uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
format_stamp: "Formatted at 2026-05-04 19:59:07 on dist-test-slave-3x8x"
I20260504 19:59:07.763015 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260504 19:59:07.766995 11782 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:07.768024 11536 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260504 19:59:07.768362 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-2-root
uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
format_stamp: "Formatted at 2026-05-04 19:59:07 on dist-test-slave-3x8x"
I20260504 19:59:07.768729 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777924745039007-11536-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 19:59:07.790622 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 19:59:07.792569 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 19:59:07.795894 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 19:59:07.799314 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 19:59:07.799540 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:07.799851 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 19:59:07.800001 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:08.056578 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.3:34783
I20260504 19:59:08.057346 11844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.3:34783 every 8 connection(s)
I20260504 19:59:08.089399 11845 heartbeater.cc:344] Connected to a master server at 127.11.68.62:40525
I20260504 19:59:08.090063 11845 heartbeater.cc:461] Registering TS with master...
I20260504 19:59:08.091259 11845 heartbeater.cc:507] Master 127.11.68.62:40525 requested a full tablet report, sending...
I20260504 19:59:08.094310 11568 ts_manager.cc:194] Registered new tserver with Master: 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:08.094563 11536 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019541324s
I20260504 19:59:08.096722 11568 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45934
I20260504 19:59:08.181036 11568 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45942:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 19:59:08.186440 11568 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 19:59:08.398564 11696 heartbeater.cc:499] Master 127.11.68.62:40525 was elected leader, sending a full tablet report...
I20260504 19:59:08.399748 11734 tablet_service.cc:1511] Processing CreateTablet for tablet 08cff134f57545aea6a4360adb6b26f4 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260504 19:59:08.407475 11735 tablet_service.cc:1511] Processing CreateTablet for tablet 3317460564e84b8a85fe4b6d6613223e (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260504 19:59:08.435901 11809 tablet_service.cc:1511] Processing CreateTablet for tablet 3317460564e84b8a85fe4b6d6613223e (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260504 19:59:08.424088 11810 tablet_service.cc:1511] Processing CreateTablet for tablet 3d5d140270334e75b8e7d71189cced30 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260504 19:59:08.408468 11736 tablet_service.cc:1511] Processing CreateTablet for tablet 3d5d140270334e75b8e7d71189cced30 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260504 19:59:08.444896 11810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5d140270334e75b8e7d71189cced30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.448139 11734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cff134f57545aea6a4360adb6b26f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.465839 11733 tablet_service.cc:1511] Processing CreateTablet for tablet 9928836addc24622ab5936846785460b (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260504 19:59:08.463284 11736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5d140270334e75b8e7d71189cced30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.469303 11733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9928836addc24622ab5936846785460b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.464768 11809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3317460564e84b8a85fe4b6d6613223e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.463634 11807 tablet_service.cc:1511] Processing CreateTablet for tablet 9928836addc24622ab5936846785460b (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260504 19:59:08.471197 11808 tablet_service.cc:1511] Processing CreateTablet for tablet 08cff134f57545aea6a4360adb6b26f4 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260504 19:59:08.478276 11731 tablet_service.cc:1511] Processing CreateTablet for tablet ac981d4f70854ed184e117453b208344 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260504 19:59:08.464138 11732 tablet_service.cc:1511] Processing CreateTablet for tablet 916ddba7631c4852add1ea09e44294d5 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260504 19:59:08.474045 11806 tablet_service.cc:1511] Processing CreateTablet for tablet 916ddba7631c4852add1ea09e44294d5 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260504 19:59:08.483057 11807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9928836addc24622ab5936846785460b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.487422 11808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cff134f57545aea6a4360adb6b26f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.490779 11735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3317460564e84b8a85fe4b6d6613223e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.494016 11805 tablet_service.cc:1511] Processing CreateTablet for tablet ac981d4f70854ed184e117453b208344 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260504 19:59:08.498461 11806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 916ddba7631c4852add1ea09e44294d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.497794 11731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac981d4f70854ed184e117453b208344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.506084 11805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac981d4f70854ed184e117453b208344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.498389 11730 tablet_service.cc:1511] Processing CreateTablet for tablet df7a5d6a6747434abed7c2460fe3d6fb (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260504 19:59:08.509692 11804 tablet_service.cc:1511] Processing CreateTablet for tablet df7a5d6a6747434abed7c2460fe3d6fb (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260504 19:59:08.516597 11732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 916ddba7631c4852add1ea09e44294d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.521072 11804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7a5d6a6747434abed7c2460fe3d6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.532393 11730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7a5d6a6747434abed7c2460fe3d6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.547259 11661 tablet_service.cc:1511] Processing CreateTablet for tablet 3d5d140270334e75b8e7d71189cced30 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260504 19:59:08.534282 11660 tablet_service.cc:1511] Processing CreateTablet for tablet 3317460564e84b8a85fe4b6d6613223e (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260504 19:59:08.537426 11659 tablet_service.cc:1511] Processing CreateTablet for tablet 08cff134f57545aea6a4360adb6b26f4 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260504 19:59:08.566174 11657 tablet_service.cc:1511] Processing CreateTablet for tablet 916ddba7631c4852add1ea09e44294d5 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260504 19:59:08.573295 11661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d5d140270334e75b8e7d71189cced30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.565034 11656 tablet_service.cc:1511] Processing CreateTablet for tablet ac981d4f70854ed184e117453b208344 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260504 19:59:08.583521 11654 tablet_service.cc:1511] Processing CreateTablet for tablet aa94336f31f248dfb81d211d4d4c8e3e (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260504 19:59:08.596097 11729 tablet_service.cc:1511] Processing CreateTablet for tablet aa94336f31f248dfb81d211d4d4c8e3e (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260504 19:59:08.606266 11729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa94336f31f248dfb81d211d4d4c8e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.566249 11658 tablet_service.cc:1511] Processing CreateTablet for tablet 9928836addc24622ab5936846785460b (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260504 19:59:08.573311 11655 tablet_service.cc:1511] Processing CreateTablet for tablet df7a5d6a6747434abed7c2460fe3d6fb (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260504 19:59:08.625535 11660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3317460564e84b8a85fe4b6d6613223e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.633523 11656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac981d4f70854ed184e117453b208344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.650691 11658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9928836addc24622ab5936846785460b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.664769 11654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa94336f31f248dfb81d211d4d4c8e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.669462 11655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7a5d6a6747434abed7c2460fe3d6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.675657 11803 tablet_service.cc:1511] Processing CreateTablet for tablet aa94336f31f248dfb81d211d4d4c8e3e (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260504 19:59:08.680410 11803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa94336f31f248dfb81d211d4d4c8e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.707672 11659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cff134f57545aea6a4360adb6b26f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.719866 11657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 916ddba7631c4852add1ea09e44294d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.744110 11771 heartbeater.cc:499] Master 127.11.68.62:40525 was elected leader, sending a full tablet report...
I20260504 19:59:08.760051 11653 tablet_service.cc:1511] Processing CreateTablet for tablet decedeb8ba0a497ea6180094250bcc98 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260504 19:59:08.768364 11653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet decedeb8ba0a497ea6180094250bcc98. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 19:59:08.811357 11616 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 19:59:08.822906 11728 tablet_service.cc:1511] Processing CreateTablet for tablet decedeb8ba0a497ea6180094250bcc98 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260504 19:59:08.824409 11728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet decedeb8ba0a497ea6180094250bcc98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.832099 11805 tablet_service.cc:1511] Processing CreateTablet for tablet decedeb8ba0a497ea6180094250bcc98 (DEFAULT_TABLE table=test-workload [id=b6e202cadf0b419d8b9a38b23993b62b]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260504 19:59:08.837505 11805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet decedeb8ba0a497ea6180094250bcc98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:08.851840 11865 tablet_bootstrap.cc:492] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:08.877151 11864 tablet_bootstrap.cc:492] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
I20260504 19:59:08.927489 11864 tablet_bootstrap.cc:654] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:08.940119 11865 tablet_bootstrap.cc:654] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:08.956995 11866 tablet_bootstrap.cc:492] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:08.992018 11865 tablet_bootstrap.cc:492] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:08.992264 11864 tablet_bootstrap.cc:492] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:08.992992 11865 ts_tablet_manager.cc:1403] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.160s user 0.017s sys 0.018s
I20260504 19:59:08.993140 11864 ts_tablet_manager.cc:1403] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.162s user 0.009s sys 0.022s
I20260504 19:59:09.001216 11866 tablet_bootstrap.cc:654] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.001754 11865 raft_consensus.cc:359] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.001865 11864 raft_consensus.cc:359] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.003075 11865 raft_consensus.cc:385] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.003123 11864 raft_consensus.cc:385] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.003815 11864 raft_consensus.cc:740] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.003760 11865 raft_consensus.cc:740] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.005419 11864 consensus_queue.cc:260] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.005515 11865 consensus_queue.cc:260] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.018049 11864 ts_tablet_manager.cc:1434] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.024s user 0.020s sys 0.001s
I20260504 19:59:09.019572 11864 tablet_bootstrap.cc:492] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
I20260504 19:59:09.019636 11866 tablet_bootstrap.cc:492] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:09.019747 11865 ts_tablet_manager.cc:1434] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.026s user 0.001s sys 0.021s
I20260504 19:59:09.020438 11866 ts_tablet_manager.cc:1403] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.020s
I20260504 19:59:09.021796 11845 heartbeater.cc:499] Master 127.11.68.62:40525 was elected leader, sending a full tablet report...
I20260504 19:59:09.022333 11865 tablet_bootstrap.cc:492] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:09.036567 11864 tablet_bootstrap.cc:654] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.036826 11866 raft_consensus.cc:359] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.037998 11866 raft_consensus.cc:385] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.038583 11866 raft_consensus.cc:740] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.039988 11866 consensus_queue.cc:260] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.050338 11865 tablet_bootstrap.cc:654] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.059957 11866 ts_tablet_manager.cc:1434] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.039s user 0.003s sys 0.010s
I20260504 19:59:09.061784 11866 tablet_bootstrap.cc:492] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:09.073128 11864 tablet_bootstrap.cc:492] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:09.074015 11864 ts_tablet_manager.cc:1403] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.055s user 0.016s sys 0.034s
I20260504 19:59:09.083178 11864 raft_consensus.cc:359] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.084347 11864 raft_consensus.cc:385] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.084865 11864 raft_consensus.cc:740] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.086010 11864 consensus_queue.cc:260] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.089679 11865 tablet_bootstrap.cc:492] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:09.090575 11865 ts_tablet_manager.cc:1403] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.031s
I20260504 19:59:09.092111 11864 ts_tablet_manager.cc:1434] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.018s user 0.010s sys 0.002s
I20260504 19:59:09.093606 11864 tablet_bootstrap.cc:492] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
I20260504 19:59:09.096442 11865 raft_consensus.cc:359] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.100642 11865 raft_consensus.cc:385] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.101198 11865 raft_consensus.cc:740] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.100086 11866 tablet_bootstrap.cc:654] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.103590 11865 consensus_queue.cc:260] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.109390 11864 tablet_bootstrap.cc:654] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.109403 11865 ts_tablet_manager.cc:1434] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.018s user 0.005s sys 0.006s
I20260504 19:59:09.112294 11865 tablet_bootstrap.cc:492] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:09.118407 11870 raft_consensus.cc:493] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.119072 11870 raft_consensus.cc:515] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.125860 11870 leader_election.cc:290] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.133774 11866 tablet_bootstrap.cc:492] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:09.134729 11866 ts_tablet_manager.cc:1403] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.012s
I20260504 19:59:09.159754 11865 tablet_bootstrap.cc:654] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.154002 11870 raft_consensus.cc:493] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.163789 11870 raft_consensus.cc:515] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.177495 11870 leader_election.cc:290] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.180365 11871 raft_consensus.cc:493] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.182458 11871 raft_consensus.cc:515] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.192489 11866 raft_consensus.cc:359] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.195680 11866 raft_consensus.cc:385] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.195776 11871 leader_election.cc:290] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:09.196291 11866 raft_consensus.cc:740] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.201124 11866 consensus_queue.cc:260] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.209559 11864 tablet_bootstrap.cc:492] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:09.210649 11864 ts_tablet_manager.cc:1403] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.117s user 0.027s sys 0.024s
I20260504 19:59:09.226384 11864 raft_consensus.cc:359] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.234127 11864 raft_consensus.cc:385] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.235741 11864 raft_consensus.cc:740] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.237018 11864 consensus_queue.cc:260] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.258190 11866 ts_tablet_manager.cc:1434] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.123s user 0.014s sys 0.004s
I20260504 19:59:09.258992 11864 ts_tablet_manager.cc:1434] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.048s user 0.006s sys 0.007s
I20260504 19:59:09.260003 11866 tablet_bootstrap.cc:492] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:09.260520 11864 tablet_bootstrap.cc:492] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
I20260504 19:59:09.267733 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
I20260504 19:59:09.281049 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
I20260504 19:59:09.286689 11820 raft_consensus.cc:2468] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 0.
I20260504 19:59:09.288767 11865 tablet_bootstrap.cc:492] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:09.302204 11871 raft_consensus.cc:493] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.307047 11873 raft_consensus.cc:493] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.305892 11871 raft_consensus.cc:515] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.307749 11873 raft_consensus.cc:515] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.310549 11871 leader_election.cc:290] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:09.305985 11865 ts_tablet_manager.cc:1403] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.194s user 0.028s sys 0.056s
I20260504 19:59:09.314563 11873 leader_election.cc:290] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.293190 11712 leader_election.cc:304] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:09.332067 11870 raft_consensus.cc:2804] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:09.333177 11870 raft_consensus.cc:493] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:09.334144 11870 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
W20260504 19:59:09.338469 11712 leader_election.cc:343] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.366532 11865 raft_consensus.cc:359] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:09.368402 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
I20260504 19:59:09.371771 11865 raft_consensus.cc:385] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.375375 11865 raft_consensus.cc:740] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.381196 11870 raft_consensus.cc:515] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.376832 11865 consensus_queue.cc:260] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:09.384079 11864 tablet_bootstrap.cc:654] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.384214 11746 raft_consensus.cc:2393] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in current term 1: Already voted for candidate 50b6b0a3f3224c01a95509c61e631d12 in this term.
I20260504 19:59:09.386371 11745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac981d4f70854ed184e117453b208344" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
I20260504 19:59:09.392678 11671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:09.397421 11870 leader_election.cc:290] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.397449 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
W20260504 19:59:09.402598 11785 leader_election.cc:343] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.403959 11865 ts_tablet_manager.cc:1434] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.090s user 0.007s sys 0.007s
I20260504 19:59:09.400169 11670 raft_consensus.cc:2468] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 0.
I20260504 19:59:09.405025 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:09.407042 11819 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.407872 11865 tablet_bootstrap.cc:492] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:09.407441 11671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4"
W20260504 19:59:09.410821 11709 leader_election.cc:343] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.411561 11866 tablet_bootstrap.cc:654] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.413142 11709 leader_election.cc:304] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600
W20260504 19:59:09.415350 11709 leader_election.cc:343] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.416816 11870 raft_consensus.cc:2804] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:09.417443 11870 raft_consensus.cc:493] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:09.418131 11870 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.442222 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
W20260504 19:59:09.447314 11637 leader_election.cc:343] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.463388 11864 tablet_bootstrap.cc:492] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:09.464903 11864 ts_tablet_manager.cc:1403] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.205s user 0.021s sys 0.022s
I20260504 19:59:09.471813 11671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:09.472410 11864 raft_consensus.cc:359] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.473913 11864 raft_consensus.cc:385] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.474231 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac981d4f70854ed184e117453b208344" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:09.474351 11864 raft_consensus.cc:740] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.476843 11864 consensus_queue.cc:260] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.477708 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
W20260504 19:59:09.479653 11783 leader_election.cc:343] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.479332 11746 raft_consensus.cc:2468] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 0.
I20260504 19:59:09.480643 11783 leader_election.cc:304] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4
I20260504 19:59:09.486667 11870 raft_consensus.cc:515] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.487223 11636 leader_election.cc:304] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:09.488138 11819 raft_consensus.cc:2468] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 1.
I20260504 19:59:09.491703 11712 leader_election.cc:304] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: f65745628d11455ab10c4c37fc004db4
I20260504 19:59:09.493484 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:09.492558 11873 raft_consensus.cc:2804] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:09.494330 11865 tablet_bootstrap.cc:654] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.494376 11670 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.494417 11873 raft_consensus.cc:493] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:09.496065 11871 raft_consensus.cc:2749] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:09.495275 11873 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.497628 11870 leader_election.cc:290] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.501080 11864 ts_tablet_manager.cc:1434] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.036s user 0.016s sys 0.003s
I20260504 19:59:09.502926 11864 tablet_bootstrap.cc:492] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
W20260504 19:59:09.503727 11783 leader_election.cc:343] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.505676 11783 leader_election.cc:304] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4
I20260504 19:59:09.504769 11870 raft_consensus.cc:2804] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:09.507324 11871 raft_consensus.cc:2749] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:09.511096 11870 raft_consensus.cc:697] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:09.511435 11866 tablet_bootstrap.cc:492] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:09.514335 11866 ts_tablet_manager.cc:1403] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.255s user 0.016s sys 0.031s
I20260504 19:59:09.512437 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
W20260504 19:59:09.524436 11712 leader_election.cc:343] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.523950 11866 raft_consensus.cc:359] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.514752 11870 consensus_queue.cc:237] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.527182 11866 raft_consensus.cc:385] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.532749 11865 tablet_bootstrap.cc:492] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:09.534163 11865 ts_tablet_manager.cc:1403] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.127s user 0.015s sys 0.018s
I20260504 19:59:09.528398 11866 raft_consensus.cc:740] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.535544 11866 consensus_queue.cc:260] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.549398 11866 ts_tablet_manager.cc:1434] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.035s user 0.010s sys 0.010s
I20260504 19:59:09.551107 11566 catalog_manager.cc:5671] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:09.554731 11866 tablet_bootstrap.cc:492] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:09.564463 11864 tablet_bootstrap.cc:654] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.567836 11865 raft_consensus.cc:359] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.569288 11865 raft_consensus.cc:385] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.569995 11865 raft_consensus.cc:740] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.582551 11866 tablet_bootstrap.cc:654] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.584743 11873 raft_consensus.cc:515] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.591735 11865 consensus_queue.cc:260] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.593117 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:09.595510 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:09.596339 11746 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
W20260504 19:59:09.600343 11637 leader_election.cc:343] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.604215 11873 leader_election.cc:290] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.605659 11865 ts_tablet_manager.cc:1434] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.067s user 0.010s sys 0.002s
I20260504 19:59:09.607163 11865 tablet_bootstrap.cc:492] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:09.617430 11864 tablet_bootstrap.cc:492] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:09.618777 11864 ts_tablet_manager.cc:1403] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.117s user 0.017s sys 0.012s
I20260504 19:59:09.630484 11866 tablet_bootstrap.cc:492] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:09.633164 11865 tablet_bootstrap.cc:654] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.635874 11866 ts_tablet_manager.cc:1403] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.081s user 0.018s sys 0.013s
I20260504 19:59:09.640486 11864 raft_consensus.cc:359] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.641856 11864 raft_consensus.cc:385] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.642401 11864 raft_consensus.cc:740] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.644070 11864 consensus_queue.cc:260] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.648487 11866 raft_consensus.cc:359] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.649202 11864 ts_tablet_manager.cc:1434] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.030s user 0.010s sys 0.004s
I20260504 19:59:09.649969 11866 raft_consensus.cc:385] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.650616 11866 raft_consensus.cc:740] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.651319 11864 tablet_bootstrap.cc:492] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
I20260504 19:59:09.652438 11866 consensus_queue.cc:260] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.661115 11866 ts_tablet_manager.cc:1434] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.023s user 0.009s sys 0.002s
I20260504 19:59:09.662668 11866 tablet_bootstrap.cc:492] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:09.664403 11865 tablet_bootstrap.cc:492] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:09.665180 11865 ts_tablet_manager.cc:1403] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.027s
I20260504 19:59:09.672748 11865 raft_consensus.cc:359] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.674604 11865 raft_consensus.cc:385] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.675148 11865 raft_consensus.cc:740] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.676683 11865 consensus_queue.cc:260] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.681449 11865 ts_tablet_manager.cc:1434] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
I20260504 19:59:09.683157 11865 tablet_bootstrap.cc:492] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:09.683324 11864 tablet_bootstrap.cc:654] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.685768 11670 raft_consensus.cc:2468] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 1.
I20260504 19:59:09.687264 11746 raft_consensus.cc:2468] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 1.
I20260504 19:59:09.689787 11636 leader_election.cc:304] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:09.690263 11709 leader_election.cc:304] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:09.692054 11870 raft_consensus.cc:2804] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:09.692597 11873 raft_consensus.cc:2804] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:09.693087 11870 raft_consensus.cc:697] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:09.693645 11873 raft_consensus.cc:697] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:09.695065 11870 consensus_queue.cc:237] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.695474 11873 consensus_queue.cc:237] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.699250 11865 tablet_bootstrap.cc:654] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.735729 11567 catalog_manager.cc:5671] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:09.750689 11866 tablet_bootstrap.cc:654] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.753955 11864 tablet_bootstrap.cc:492] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:09.754720 11864 ts_tablet_manager.cc:1403] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.104s user 0.024s sys 0.012s
I20260504 19:59:09.753516 11568 catalog_manager.cc:5671] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:09.755630 11871 raft_consensus.cc:493] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.756320 11871 raft_consensus.cc:515] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.758766 11865 tablet_bootstrap.cc:492] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:09.763199 11865 ts_tablet_manager.cc:1403] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.080s user 0.019s sys 0.020s
I20260504 19:59:09.769500 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:09.769309 11865 raft_consensus.cc:359] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.771476 11865 raft_consensus.cc:385] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.772644 11865 raft_consensus.cc:740] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.771770 11871 leader_election.cc:290] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
W20260504 19:59:09.771978 11783 leader_election.cc:343] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.770820 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
I20260504 19:59:09.774941 11746 raft_consensus.cc:2468] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 0.
I20260504 19:59:09.775938 11870 raft_consensus.cc:493] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.775104 11865 consensus_queue.cc:260] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.776532 11870 raft_consensus.cc:515] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.778210 11785 leader_election.cc:304] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: f65745628d11455ab10c4c37fc004db4
I20260504 19:59:09.779913 11871 raft_consensus.cc:2804] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:09.785450 11871 raft_consensus.cc:493] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:09.786175 11871 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.790319 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
I20260504 19:59:09.791555 11870 leader_election.cc:290] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.784348 11864 raft_consensus.cc:359] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.794804 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:09.798558 11864 raft_consensus.cc:385] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.799376 11864 raft_consensus.cc:740] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
W20260504 19:59:09.801075 11709 leader_election.cc:343] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.800617 11864 consensus_queue.cc:260] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.806200 11864 ts_tablet_manager.cc:1434] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.051s user 0.014s sys 0.002s
I20260504 19:59:09.808025 11864 tablet_bootstrap.cc:492] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
W20260504 19:59:09.816643 11616 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 19:59:09.817885 11865 ts_tablet_manager.cc:1434] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.052s user 0.007s sys 0.007s
I20260504 19:59:09.819454 11865 tablet_bootstrap.cc:492] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:09.823011 11866 tablet_bootstrap.cc:492] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:09.825547 11866 ts_tablet_manager.cc:1403] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.163s user 0.024s sys 0.003s
I20260504 19:59:09.833379 11865 tablet_bootstrap.cc:654] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.816515 11909 raft_consensus.cc:493] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.846174 11909 raft_consensus.cc:515] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.849936 11909 leader_election.cc:290] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:09.854663 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
I20260504 19:59:09.855077 11864 tablet_bootstrap.cc:654] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.855752 11746 raft_consensus.cc:2468] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 0.
I20260504 19:59:09.857403 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:09.857537 11785 leader_election.cc:304] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:09.859802 11873 raft_consensus.cc:493] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 19:59:09.860456 11783 leader_election.cc:343] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.861510 11909 raft_consensus.cc:2804] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:09.860543 11873 raft_consensus.cc:515] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.864178 11909 raft_consensus.cc:493] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:09.864593 11865 tablet_bootstrap.cc:492] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:09.864830 11909 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.865430 11865 ts_tablet_manager.cc:1403] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.019s
I20260504 19:59:09.846705 11866 raft_consensus.cc:359] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:09.868899 11866 raft_consensus.cc:385] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.869381 11866 raft_consensus.cc:740] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.868575 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac981d4f70854ed184e117453b208344" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
W20260504 19:59:09.872373 11636 leader_election.cc:343] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.872417 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac981d4f70854ed184e117453b208344" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
I20260504 19:59:09.873374 11873 leader_election.cc:290] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.873358 11820 raft_consensus.cc:2468] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 0.
I20260504 19:59:09.873443 11865 raft_consensus.cc:359] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.874745 11865 raft_consensus.cc:385] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.875182 11865 raft_consensus.cc:740] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.875490 11637 leader_election.cc:304] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters: 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:09.876858 11873 raft_consensus.cc:2804] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:09.877312 11873 raft_consensus.cc:493] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:09.878019 11873 raft_consensus.cc:3060] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.880970 11864 tablet_bootstrap.cc:492] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:09.881647 11864 ts_tablet_manager.cc:1403] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.011s
I20260504 19:59:09.876211 11865 consensus_queue.cc:260] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.870525 11866 consensus_queue.cc:260] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:09.892416 11864 raft_consensus.cc:359] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.893432 11864 raft_consensus.cc:385] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:09.893893 11864 raft_consensus.cc:740] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
I20260504 19:59:09.894922 11864 consensus_queue.cc:260] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.889492 11871 raft_consensus.cc:515] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.900322 11864 ts_tablet_manager.cc:1434] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.018s user 0.008s sys 0.001s
I20260504 19:59:09.901914 11864 tablet_bootstrap.cc:492] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
I20260504 19:59:09.903337 11866 ts_tablet_manager.cc:1434] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.073s user 0.011s sys 0.000s
I20260504 19:59:09.903914 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:09.904735 11746 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.906606 11819 raft_consensus.cc:2393] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50b6b0a3f3224c01a95509c61e631d12 in current term 1: Already voted for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in this term.
I20260504 19:59:09.910985 11712 leader_election.cc:304] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12; no voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4
I20260504 19:59:09.913093 11895 raft_consensus.cc:493] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.913781 11895 raft_consensus.cc:515] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:09.914589 11871 leader_election.cc:290] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:09.915607 11871 raft_consensus.cc:493] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.916365 11871 raft_consensus.cc:515] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:09.918200 11745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
W20260504 19:59:09.920317 11636 leader_election.cc:343] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.920423 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:09.922024 11895 leader_election.cc:290] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:09.925045 11864 tablet_bootstrap.cc:654] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.930370 11866 tablet_bootstrap.cc:492] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:09.934620 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
W20260504 19:59:09.936079 11783 leader_election.cc:343] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:09.945129 11819 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 0.
I20260504 19:59:09.961457 11871 leader_election.cc:290] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535)
I20260504 19:59:09.957785 11637 leader_election.cc:304] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters: 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:09.965166 11913 raft_consensus.cc:493] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:09.967152 11864 tablet_bootstrap.cc:492] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:09.967103 11745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
I20260504 19:59:09.969288 11864 ts_tablet_manager.cc:1403] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.015s
I20260504 19:59:09.969678 11865 ts_tablet_manager.cc:1434] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.104s user 0.017s sys 0.018s
I20260504 19:59:09.964620 11895 raft_consensus.cc:2804] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:09.972987 11895 raft_consensus.cc:493] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:09.973383 11895 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.965785 11913 raft_consensus.cc:515] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.979432 11913 leader_election.cc:290] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:09.980397 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:09.981451 11865 tablet_bootstrap.cc:492] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:09.982115 11866 tablet_bootstrap.cc:654] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.983196 11669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa94336f31f248dfb81d211d4d4c8e3e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
W20260504 19:59:09.974229 11785 leader_election.cc:343] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Illegal state: must be running to vote when last-logged opid is not known
W20260504 19:59:09.992415 11783 leader_election.cc:343] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
W20260504 19:59:09.983572 11709 consensus_peers.cc:597] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 -> Peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Couldn't send request to peer f65745628d11455ab10c4c37fc004db4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 19:59:09.984968 11909 raft_consensus.cc:515] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:09.994357 11745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa94336f31f248dfb81d211d4d4c8e3e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
I20260504 19:59:09.995431 11745 raft_consensus.cc:2468] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 0.
I20260504 19:59:09.996416 11909 leader_election.cc:290] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:09.998090 11785 leader_election.cc:304] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: f65745628d11455ab10c4c37fc004db4
I20260504 19:59:10.000398 11745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:09.985647 11873 raft_consensus.cc:515] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.001565 11745 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:09.985598 11864 raft_consensus.cc:359] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:09.998901 11865 tablet_bootstrap.cc:654] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:09.986950 11746 raft_consensus.cc:2468] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 1.
I20260504 19:59:10.005234 11870 raft_consensus.cc:2749] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:10.005470 11909 raft_consensus.cc:2804] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:10.004530 11864 raft_consensus.cc:385] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:10.005956 11909 raft_consensus.cc:493] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:10.006191 11864 raft_consensus.cc:740] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
I20260504 19:59:10.007063 11864 consensus_queue.cc:260] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:10.007755 11785 leader_election.cc:304] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: f65745628d11455ab10c4c37fc004db4
I20260504 19:59:10.009214 11913 raft_consensus.cc:2804] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:10.006376 11909 raft_consensus.cc:3060] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.009465 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac981d4f70854ed184e117453b208344" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:10.011921 11913 raft_consensus.cc:697] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Becoming Leader. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:10.018550 11746 raft_consensus.cc:3060] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.028165 11913 consensus_queue.cc:237] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.029913 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac981d4f70854ed184e117453b208344" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:10.030834 11819 raft_consensus.cc:3060] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.035128 11864 ts_tablet_manager.cc:1434] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.066s user 0.009s sys 0.016s
I20260504 19:59:10.036466 11864 tablet_bootstrap.cc:492] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12: Bootstrap starting.
I20260504 19:59:10.032537 11873 leader_election.cc:290] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:10.034960 11669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4"
W20260504 19:59:10.050693 11783 leader_election.cc:343] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:10.045424 11819 raft_consensus.cc:2468] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 1.
I20260504 19:59:10.051065 11895 raft_consensus.cc:515] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:10.054570 11670 raft_consensus.cc:2393] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in current term 1: Already voted for candidate f65745628d11455ab10c4c37fc004db4 in this term.
I20260504 19:59:10.057411 11895 leader_election.cc:290] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:10.055141 11637 leader_election.cc:304] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:10.060127 11746 raft_consensus.cc:2468] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 1.
I20260504 19:59:10.062866 11909 raft_consensus.cc:515] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.064884 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:10.065238 11866 tablet_bootstrap.cc:492] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:10.065762 11819 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.066027 11866 ts_tablet_manager.cc:1403] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.136s user 0.019s sys 0.006s
I20260504 19:59:10.067876 11783 leader_election.cc:304] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4
I20260504 19:59:10.065164 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:10.069612 11909 leader_election.cc:290] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:10.071691 11865 tablet_bootstrap.cc:492] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:10.069729 11744 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.069056 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa94336f31f248dfb81d211d4d4c8e3e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:10.072149 11865 ts_tablet_manager.cc:1403] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.093s user 0.012s sys 0.014s
I20260504 19:59:10.072835 11746 raft_consensus.cc:3060] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.073343 11895 raft_consensus.cc:2804] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:10.079645 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa94336f31f248dfb81d211d4d4c8e3e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:10.080066 11895 raft_consensus.cc:697] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:10.079128 11864 tablet_bootstrap.cc:654] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:10.079514 11745 raft_consensus.cc:2468] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 1.
I20260504 19:59:10.084658 11865 raft_consensus.cc:359] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.085834 11865 raft_consensus.cc:385] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:10.076988 11866 raft_consensus.cc:359] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.086236 11865 raft_consensus.cc:740] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:10.082693 11895 consensus_queue.cc:237] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.087190 11865 consensus_queue.cc:260] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.091923 11870 consensus_queue.cc:1048] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.088279 11866 raft_consensus.cc:385] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:10.095745 11866 raft_consensus.cc:740] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:10.084759 11565 catalog_manager.cc:5671] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:10.097020 11785 leader_election.cc:304] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: f65745628d11455ab10c4c37fc004db4
I20260504 19:59:10.098553 11913 raft_consensus.cc:2804] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:10.099213 11913 raft_consensus.cc:697] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Becoming Leader. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:10.097177 11866 consensus_queue.cc:260] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.100405 11913 consensus_queue.cc:237] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.103340 11865 ts_tablet_manager.cc:1434] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.030s user 0.010s sys 0.000s
I20260504 19:59:10.113709 11865 tablet_bootstrap.cc:492] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Bootstrap starting.
I20260504 19:59:10.116338 11670 raft_consensus.cc:3060] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.113916 11744 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 1.
I20260504 19:59:10.124087 11819 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 1.
I20260504 19:59:10.122423 11873 raft_consensus.cc:493] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:10.130007 11873 raft_consensus.cc:515] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.134819 11745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
W20260504 19:59:10.137320 11636 leader_election.cc:343] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:10.138873 11873 leader_election.cc:290] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:10.138947 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
I20260504 19:59:10.139922 11866 ts_tablet_manager.cc:1434] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.073s user 0.007s sys 0.006s
I20260504 19:59:10.140111 11909 raft_consensus.cc:2749] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:10.140105 11820 raft_consensus.cc:2468] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 0.
I20260504 19:59:10.142987 11637 leader_election.cc:304] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:10.144671 11873 raft_consensus.cc:2804] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:10.145100 11637 leader_election.cc:304] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters: 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:10.145504 11873 raft_consensus.cc:697] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:10.155790 11919 raft_consensus.cc:2804] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:10.154907 11873 consensus_queue.cc:237] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:10.156847 11919 raft_consensus.cc:493] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:10.151492 11870 consensus_queue.cc:1048] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260504 19:59:10.166281 11919 raft_consensus.cc:3060] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.163403 11866 tablet_bootstrap.cc:492] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:10.174629 11913 raft_consensus.cc:493] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:10.176277 11913 raft_consensus.cc:515] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.179785 11913 leader_election.cc:290] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:10.157781 11670 raft_consensus.cc:2468] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 1.
I20260504 19:59:10.188988 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
I20260504 19:59:10.198798 11783 leader_election.cc:304] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:10.200294 11913 raft_consensus.cc:2804] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:10.201098 11913 raft_consensus.cc:697] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Becoming Leader. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:10.201875 11746 raft_consensus.cc:2468] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 1.
I20260504 19:59:10.198755 11565 catalog_manager.cc:5671] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:10.202471 11913 consensus_queue.cc:237] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.196321 11568 catalog_manager.cc:5671] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:10.219561 11865 tablet_bootstrap.cc:654] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:10.245445 11671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:10.256395 11864 tablet_bootstrap.cc:492] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12: No bootstrap required, opened a new log
I20260504 19:59:10.256896 11864 ts_tablet_manager.cc:1403] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12: Time spent bootstrapping tablet: real 0.221s user 0.006s sys 0.022s
I20260504 19:59:10.261775 11919 raft_consensus.cc:515] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.265256 11919 leader_election.cc:290] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
W20260504 19:59:10.266119 11637 consensus_peers.cc:597] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 -> Peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783): Couldn't send request to peer 7b3ddc772c494eb4a7bf6bf1ec8ea600. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260504 19:59:10.266717 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:10.267478 11783 leader_election.cc:304] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4
I20260504 19:59:10.269104 11913 raft_consensus.cc:2749] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:10.280781 11818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:10.283080 11818 raft_consensus.cc:3060] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.288250 11864 raft_consensus.cc:359] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.289319 11864 raft_consensus.cc:385] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:10.276970 11921 consensus_queue.cc:1048] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20260504 19:59:10.290815 11636 leader_election.cc:343] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:10.291366 11866 tablet_bootstrap.cc:654] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:10.289682 11864 raft_consensus.cc:740] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Initialized, Role: FOLLOWER
I20260504 19:59:10.294480 11864 consensus_queue.cc:260] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.330595 11865 tablet_bootstrap.cc:492] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600: No bootstrap required, opened a new log
I20260504 19:59:10.331153 11865 ts_tablet_manager.cc:1403] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent bootstrapping tablet: real 0.218s user 0.019s sys 0.008s
I20260504 19:59:10.342114 11818 raft_consensus.cc:2468] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 1.
I20260504 19:59:10.338207 11820 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.346446 11865 raft_consensus.cc:359] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.396576 11865 raft_consensus.cc:385] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 19:59:10.349169 11923 log.cc:927] Time spent T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Append to log took a long time: real 0.065s user 0.001s sys 0.001s
I20260504 19:59:10.397265 11865 raft_consensus.cc:740] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Initialized, Role: FOLLOWER
I20260504 19:59:10.397900 11637 leader_election.cc:304] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters: 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:10.398167 11865 consensus_queue.cc:260] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.399686 11873 raft_consensus.cc:2804] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:10.400354 11873 raft_consensus.cc:697] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:10.401441 11865 ts_tablet_manager.cc:1434] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600: Time spent starting tablet: real 0.070s user 0.018s sys 0.017s
I20260504 19:59:10.365550 11930 raft_consensus.cc:493] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:10.401386 11873 consensus_queue.cc:237] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.404695 11930 raft_consensus.cc:515] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.388913 11864 ts_tablet_manager.cc:1434] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12: Time spent starting tablet: real 0.132s user 0.006s sys 0.011s
I20260504 19:59:10.419440 11671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65745628d11455ab10c4c37fc004db4" is_pre_election: true
I20260504 19:59:10.419761 11866 tablet_bootstrap.cc:492] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:10.420768 11866 ts_tablet_manager.cc:1403] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.262s user 0.014s sys 0.021s
I20260504 19:59:10.422389 11930 leader_election.cc:290] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:10.386572 11568 catalog_manager.cc:5671] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:10.435015 11818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
I20260504 19:59:10.436421 11818 raft_consensus.cc:2393] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50b6b0a3f3224c01a95509c61e631d12 in current term 1: Already voted for candidate f65745628d11455ab10c4c37fc004db4 in this term.
I20260504 19:59:10.438221 11712 leader_election.cc:304] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12; no voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4
I20260504 19:59:10.428369 11866 raft_consensus.cc:359] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.439196 11866 raft_consensus.cc:385] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:10.439661 11866 raft_consensus.cc:740] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:10.440910 11866 consensus_queue.cc:260] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.442797 11565 catalog_manager.cc:5671] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:10.445559 11866 ts_tablet_manager.cc:1434] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.024s user 0.005s sys 0.005s
I20260504 19:59:10.440945 11892 raft_consensus.cc:3060] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.447273 11866 tablet_bootstrap.cc:492] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:10.454205 11930 consensus_queue.cc:1048] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.467340 11866 tablet_bootstrap.cc:654] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:10.469410 11892 raft_consensus.cc:2749] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:10.474241 11866 tablet_bootstrap.cc:492] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:10.474773 11866 ts_tablet_manager.cc:1403] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.013s
I20260504 19:59:10.477211 11873 raft_consensus.cc:493] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:10.477732 11873 raft_consensus.cc:515] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.482463 11866 raft_consensus.cc:359] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.483341 11866 raft_consensus.cc:385] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:10.483705 11866 raft_consensus.cc:740] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:10.482489 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" is_pre_election: true
I20260504 19:59:10.484580 11866 consensus_queue.cc:260] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.483485 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50b6b0a3f3224c01a95509c61e631d12" is_pre_election: true
I20260504 19:59:10.488178 11873 leader_election.cc:290] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:10.492544 11636 leader_election.cc:304] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f65745628d11455ab10c4c37fc004db4; no voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:10.488188 11866 ts_tablet_manager.cc:1434] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260504 19:59:10.494153 11873 raft_consensus.cc:2749] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:10.494747 11866 tablet_bootstrap.cc:492] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4: Bootstrap starting.
I20260504 19:59:10.503139 11866 tablet_bootstrap.cc:654] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:10.501890 11568 catalog_manager.cc:5671] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:10.515561 11866 tablet_bootstrap.cc:492] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4: No bootstrap required, opened a new log
I20260504 19:59:10.516067 11866 ts_tablet_manager.cc:1403] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.009s
I20260504 19:59:10.519994 11866 raft_consensus.cc:359] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.520802 11866 raft_consensus.cc:385] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:10.521119 11866 raft_consensus.cc:740] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Initialized, Role: FOLLOWER
I20260504 19:59:10.522043 11866 consensus_queue.cc:260] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:10.524214 11866 ts_tablet_manager.cc:1434] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260504 19:59:10.545094 11913 consensus_queue.cc:1048] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.564920 11671 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.582193 11913 consensus_queue.cc:1048] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.613404 11873 consensus_queue.cc:1048] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.614924 11895 consensus_queue.cc:1048] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.629230 11671 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.646629 11913 consensus_queue.cc:1048] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.649708 11913 consensus_queue.cc:1048] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.651293 11873 consensus_queue.cc:1048] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.700784 11919 consensus_queue.cc:1048] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.760766 11913 consensus_queue.cc:1048] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.765105 11909 consensus_queue.cc:1048] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.758334 11819 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:10.826622 11873 consensus_queue.cc:1048] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.914171 11820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
mode: GRACEFUL
new_leader_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:50508
I20260504 19:59:10.915186 11820 raft_consensus.cc:606] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Received request to transfer leadership to TS f65745628d11455ab10c4c37fc004db4
I20260504 19:59:10.954356 11919 consensus_queue.cc:1048] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:10.992877 11909 raft_consensus.cc:993] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: : Instructing follower f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:10.993598 11909 raft_consensus.cc:1081] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Signalling peer f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:10.996208 11671 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:55248
I20260504 19:59:10.997005 11671 raft_consensus.cc:493] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:10.997504 11671 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:10.997673 11669 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:11.013727 11671 raft_consensus.cc:515] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.018481 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:11.019250 11746 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.019616 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:11.027411 11744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
mode: GRACEFUL
new_leader_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:55162
I20260504 19:59:11.028774 11746 raft_consensus.cc:2468] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 2.
I20260504 19:59:11.029222 11744 raft_consensus.cc:606] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Received request to transfer leadership to TS f65745628d11455ab10c4c37fc004db4
I20260504 19:59:11.020310 11819 raft_consensus.cc:3055] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:11.030663 11819 raft_consensus.cc:740] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:11.031903 11636 leader_election.cc:304] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:11.031777 11819 consensus_queue.cc:260] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.032835 11919 raft_consensus.cc:2804] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:11.033102 11819 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.033341 11919 raft_consensus.cc:697] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:11.034494 11919 consensus_queue.cc:237] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.035908 11744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9928836addc24622ab5936846785460b"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
mode: GRACEFUL
new_leader_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:55162
I20260504 19:59:11.036628 11744 raft_consensus.cc:606] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Received request to transfer leadership to TS f65745628d11455ab10c4c37fc004db4
I20260504 19:59:11.043002 11671 leader_election.cc:290] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 2 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:11.043236 11818 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:50508
I20260504 19:59:11.044020 11818 raft_consensus.cc:606] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:11.052794 11930 raft_consensus.cc:993] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12: : Instructing follower f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:11.053459 11930 raft_consensus.cc:1081] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Signalling peer f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:11.059175 11819 raft_consensus.cc:2468] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 2.
I20260504 19:59:11.064262 11669 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:55236
I20260504 19:59:11.065302 11669 raft_consensus.cc:493] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:11.065936 11669 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.069468 11567 catalog_manager.cc:5671] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 1 to 2, leader changed from 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3) to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:11.081743 11669 raft_consensus.cc:515] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.082492 11671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac981d4f70854ed184e117453b208344"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55264
I20260504 19:59:11.083334 11671 raft_consensus.cc:606] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:11.084807 11669 leader_election.cc:290] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 2 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:11.086239 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:11.087574 11744 raft_consensus.cc:3055] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:11.088153 11744 raft_consensus.cc:740] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:11.091070 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:11.091802 11819 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.097424 11669 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55264
I20260504 19:59:11.098325 11669 raft_consensus.cc:606] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:11.097844 11744 consensus_queue.cc:260] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.099843 11744 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.101703 11819 raft_consensus.cc:2468] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 2.
I20260504 19:59:11.104321 11637 leader_election.cc:304] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:11.110697 11919 raft_consensus.cc:2804] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:11.111418 11919 raft_consensus.cc:697] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:11.113049 11919 consensus_queue.cc:237] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.122179 11819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aa94336f31f248dfb81d211d4d4c8e3e"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:50508
I20260504 19:59:11.123068 11819 raft_consensus.cc:606] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:11.125705 11744 raft_consensus.cc:2468] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 2.
I20260504 19:59:11.133121 11567 catalog_manager.cc:5671] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 1 to 2, leader changed from 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2) to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:11.138736 11873 raft_consensus.cc:993] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:11.138633 11669 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
mode: GRACEFUL
new_leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:55264
I20260504 19:59:11.139394 11873 raft_consensus.cc:1081] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:11.139629 11669 raft_consensus.cc:606] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Received request to transfer leadership to TS 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:11.142225 11746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac981d4f70854ed184e117453b208344"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55158
I20260504 19:59:11.142918 11746 raft_consensus.cc:493] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:11.143370 11746 raft_consensus.cc:3060] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.153012 11746 raft_consensus.cc:515] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.156890 11669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac981d4f70854ed184e117453b208344" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:11.157707 11669 raft_consensus.cc:3055] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:11.158286 11669 raft_consensus.cc:740] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:11.159277 11669 consensus_queue.cc:260] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.160733 11669 raft_consensus.cc:3060] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.163709 11746 leader_election.cc:290] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 2 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:11.165370 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac981d4f70854ed184e117453b208344" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:11.166174 11819 raft_consensus.cc:3060] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.170197 11669 raft_consensus.cc:2468] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 2.
I20260504 19:59:11.172178 11709 leader_election.cc:304] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:11.173362 11892 raft_consensus.cc:2804] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:11.173648 11819 raft_consensus.cc:2468] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 2.
I20260504 19:59:11.174190 11892 raft_consensus.cc:697] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:11.175577 11892 consensus_queue.cc:237] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.186978 11567 catalog_manager.cc:5671] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 1 to 2, leader changed from f65745628d11455ab10c4c37fc004db4 (127.11.68.1) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:11.232455 11930 raft_consensus.cc:993] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12: : Instructing follower f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:11.233031 11892 raft_consensus.cc:1081] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Signalling peer f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:11.235029 11669 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9928836addc24622ab5936846785460b"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:55236
I20260504 19:59:11.235898 11669 raft_consensus.cc:493] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:11.236446 11669 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.247519 11669 raft_consensus.cc:515] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.251518 11669 leader_election.cc:290] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 2 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:11.252365 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:11.252622 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:11.253153 11746 raft_consensus.cc:3055] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:11.253443 11819 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.253603 11746 raft_consensus.cc:740] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:11.254763 11746 consensus_queue.cc:260] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.256445 11746 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.260880 11819 raft_consensus.cc:2468] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 2.
I20260504 19:59:11.262292 11637 leader_election.cc:304] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:11.263243 11873 raft_consensus.cc:2804] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:11.263790 11873 raft_consensus.cc:697] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:11.265260 11746 raft_consensus.cc:2468] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 2.
I20260504 19:59:11.265133 11873 consensus_queue.cc:237] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.275000 11567 catalog_manager.cc:5671] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 1 to 2, leader changed from 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2) to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:11.288682 11919 raft_consensus.cc:993] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:11.289340 11873 raft_consensus.cc:1081] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:11.292040 11746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55158
I20260504 19:59:11.292639 11746 raft_consensus.cc:493] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:11.293025 11746 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.301033 11746 raft_consensus.cc:515] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.305441 11913 raft_consensus.cc:993] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:11.306355 11913 raft_consensus.cc:1081] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:11.307125 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:11.308105 11819 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.308027 11669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:11.309072 11669 raft_consensus.cc:3055] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:11.309614 11669 raft_consensus.cc:740] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:11.310245 11746 leader_election.cc:290] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 2 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:11.311038 11669 consensus_queue.cc:260] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.313572 11746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aa94336f31f248dfb81d211d4d4c8e3e"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55146
I20260504 19:59:11.313632 11669 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.314637 11746 raft_consensus.cc:493] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:11.315258 11746 raft_consensus.cc:3060] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.320055 11819 raft_consensus.cc:2468] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 2.
I20260504 19:59:11.322118 11712 leader_election.cc:304] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:11.323518 11892 raft_consensus.cc:2804] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:11.324299 11892 raft_consensus.cc:697] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:11.325230 11669 raft_consensus.cc:2468] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 2.
I20260504 19:59:11.325855 11746 raft_consensus.cc:515] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.325659 11892 consensus_queue.cc:237] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.328934 11746 leader_election.cc:290] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 2 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:11.330962 11669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa94336f31f248dfb81d211d4d4c8e3e" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:11.333949 11818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa94336f31f248dfb81d211d4d4c8e3e" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:11.333101 11669 raft_consensus.cc:3060] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.334750 11818 raft_consensus.cc:3055] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:11.334882 11919 raft_consensus.cc:993] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4: : Instructing follower 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:11.335332 11818 raft_consensus.cc:740] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:11.335661 11919 raft_consensus.cc:1081] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Signalling peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:11.336534 11818 consensus_queue.cc:260] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.338497 11818 raft_consensus.cc:3060] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.339681 11819 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:50502
I20260504 19:59:11.340583 11819 raft_consensus.cc:493] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:11.341305 11819 raft_consensus.cc:3060] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.340713 11567 catalog_manager.cc:5671] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 1 to 2, leader changed from f65745628d11455ab10c4c37fc004db4 (127.11.68.1) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:11.354943 11909 raft_consensus.cc:993] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:11.355823 11913 raft_consensus.cc:1081] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:11.358968 11746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55146
I20260504 19:59:11.359834 11746 raft_consensus.cc:493] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:11.360440 11746 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.361687 11818 raft_consensus.cc:2468] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 2.
I20260504 19:59:11.364143 11712 leader_election.cc:304] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:11.365875 11892 raft_consensus.cc:2804] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:11.366465 11892 raft_consensus.cc:697] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:11.367712 11892 consensus_queue.cc:237] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.375208 11746 raft_consensus.cc:515] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.376014 11819 raft_consensus.cc:515] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.361680 11669 raft_consensus.cc:2468] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 2.
I20260504 19:59:11.380110 11671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:11.381101 11671 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.382628 11567 catalog_manager.cc:5671] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 1 to 2, leader changed from 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:11.387604 11819 leader_election.cc:290] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 2 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:11.389200 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:11.390095 11744 raft_consensus.cc:3060] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.395097 11671 raft_consensus.cc:2468] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 2.
I20260504 19:59:11.395991 11669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "decedeb8ba0a497ea6180094250bcc98" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:11.399128 11669 raft_consensus.cc:3055] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:11.398015 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:11.399483 11669 raft_consensus.cc:740] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:11.398463 11746 leader_election.cc:290] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 2 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:11.400369 11819 raft_consensus.cc:3055] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:11.399574 11709 leader_election.cc:304] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:11.400671 11669 consensus_queue.cc:260] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.401173 11819 raft_consensus.cc:740] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:11.401577 11892 raft_consensus.cc:2804] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:11.402108 11669 raft_consensus.cc:3060] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.402263 11819 consensus_queue.cc:260] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.402166 11892 raft_consensus.cc:697] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:11.404002 11819 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:11.404281 11892 consensus_queue.cc:237] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.412467 11819 raft_consensus.cc:2468] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 2.
I20260504 19:59:11.413878 11744 raft_consensus.cc:2468] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 2.
I20260504 19:59:11.415311 11785 leader_election.cc:304] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:11.415759 11669 raft_consensus.cc:2468] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 2.
I20260504 19:59:11.416400 11909 raft_consensus.cc:2804] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:11.417037 11909 raft_consensus.cc:697] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 LEADER]: Becoming Leader. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:11.418334 11909 consensus_queue.cc:237] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:11.424345 11567 catalog_manager.cc:5671] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 1 to 2, leader changed from 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:11.431952 11566 catalog_manager.cc:5671] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 reported cstate change: term changed from 1 to 2, leader changed from f65745628d11455ab10c4c37fc004db4 (127.11.68.1) to 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:11.523456 11744 raft_consensus.cc:1275] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.524706 11873 consensus_queue.cc:1048] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.534966 11819 raft_consensus.cc:1275] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.536347 11921 consensus_queue.cc:1048] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.560199 11819 raft_consensus.cc:1275] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.561650 11892 consensus_queue.cc:1048] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.569172 11669 raft_consensus.cc:1275] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.573668 11964 consensus_queue.cc:1048] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.575390 11744 raft_consensus.cc:1275] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.577030 11921 consensus_queue.cc:1048] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.586504 11819 raft_consensus.cc:1275] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.589419 11873 consensus_queue.cc:1048] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.645635 11819 raft_consensus.cc:1275] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.647032 11921 consensus_queue.cc:1048] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.655356 11744 raft_consensus.cc:1275] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.656936 11873 consensus_queue.cc:1048] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.789572 11669 raft_consensus.cc:1275] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.791040 11964 consensus_queue.cc:1048] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.800360 11819 raft_consensus.cc:1275] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.802188 11892 consensus_queue.cc:1048] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.829270 11744 raft_consensus.cc:1275] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.830610 11909 consensus_queue.cc:1048] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.838268 11669 raft_consensus.cc:1275] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.839607 11913 consensus_queue.cc:1048] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.916090 11819 raft_consensus.cc:1275] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.917513 11892 consensus_queue.cc:1048] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:11.925866 11669 raft_consensus.cc:1275] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:11.927786 11964 consensus_queue.cc:1048] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:12.002120 11669 raft_consensus.cc:1275] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:12.003463 11964 consensus_queue.cc:1048] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:12.011775 11819 raft_consensus.cc:1275] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:12.013422 11964 consensus_queue.cc:1048] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:14.173532 11536 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20260504 19:59:14.174400 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:14.192128 11669 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
mode: GRACEFUL
new_leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:54284
I20260504 19:59:14.192687 11669 raft_consensus.cc:606] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Received request to transfer leadership to TS 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:14.203995 11744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
mode: GRACEFUL
new_leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:38476
I20260504 19:59:14.204743 11744 raft_consensus.cc:606] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Received request to transfer leadership to TS 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:14.206126 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260504 19:59:14.206583 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:14.593427 11921 raft_consensus.cc:993] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4: : Instructing follower 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:14.594033 11980 raft_consensus.cc:1081] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Signalling peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:14.595705 11818 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:50502
I20260504 19:59:14.596280 11818 raft_consensus.cc:493] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:14.596642 11818 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:14.601361 11818 raft_consensus.cc:515] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:14.603075 11818 leader_election.cc:290] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 2 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535)
I20260504 19:59:14.604197 11669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:14.604197 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:14.605062 11669 raft_consensus.cc:3055] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:14.605090 11744 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:14.605512 11669 raft_consensus.cc:740] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:14.606395 11669 consensus_queue.cc:260] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:14.607357 11669 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:14.610154 11744 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 2.
I20260504 19:59:14.611402 11785 leader_election.cc:304] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:14.612170 11913 raft_consensus.cc:2804] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:14.612493 11669 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 2.
I20260504 19:59:14.613353 11913 raft_consensus.cc:697] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 LEADER]: Becoming Leader. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:14.614506 11913 consensus_queue.cc:237] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:14.624011 11567 catalog_manager.cc:5671] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 reported cstate change: term changed from 1 to 2, leader changed from f65745628d11455ab10c4c37fc004db4 (127.11.68.1) to 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:14.715843 11964 raft_consensus.cc:993] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12: : Instructing follower 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:14.716418 11987 raft_consensus.cc:1081] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Signalling peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:14.718359 11818 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:50486
I20260504 19:59:14.718910 11818 raft_consensus.cc:493] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:14.719264 11818 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:14.725775 11818 raft_consensus.cc:515] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:14.728001 11818 leader_election.cc:290] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 3 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:14.729122 11669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:14.729403 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:14.729921 11669 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:14.730134 11744 raft_consensus.cc:3055] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:14.730607 11744 raft_consensus.cc:740] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:14.731658 11744 consensus_queue.cc:260] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:14.733289 11744 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:14.738798 11669 raft_consensus.cc:2468] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 3.
I20260504 19:59:14.740660 11783 leader_election.cc:304] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:14.741633 11913 raft_consensus.cc:2804] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:14.742377 11913 raft_consensus.cc:697] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 LEADER]: Becoming Leader. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:14.742770 11744 raft_consensus.cc:2468] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 3.
I20260504 19:59:14.743619 11913 consensus_queue.cc:237] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:14.751366 11567 catalog_manager.cc:5671] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 reported cstate change: term changed from 2 to 3, leader changed from 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2) to 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:15.059168 11744 raft_consensus.cc:1275] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:15.060667 11913 consensus_queue.cc:1048] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:15.075897 11669 raft_consensus.cc:1275] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:15.079763 11992 consensus_queue.cc:1048] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 19:59:15.204406 11744 raft_consensus.cc:1275] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:15.205870 11992 consensus_queue.cc:1048] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:15.207602 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:15.216918 11669 raft_consensus.cc:1275] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:15.219631 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:15.219653 11913 consensus_queue.cc:1048] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 19:59:15.220500 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:16.222009 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:16.227367 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:16.227775 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:17.228752 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:17.233739 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:17.234302 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:18.235195 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:18.240407 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:18.240823 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:19.241775 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:19.247783 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:19.248236 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:20.249261 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:20.255434 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:20.255825 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:21.338737 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:21.372042 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:21.372404 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260504 19:59:22.370990 11599 debug-util.cc:398] Leaking SignalData structure 0x7b0800061060 after lost signal to thread 11539
W20260504 19:59:22.371832 11599 debug-util.cc:398] Leaking SignalData structure 0x7b0800061740 after lost signal to thread 11602
W20260504 19:59:22.372655 11599 debug-util.cc:398] Leaking SignalData structure 0x7b0800029660 after lost signal to thread 11695
I20260504 19:59:22.373309 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260504 19:59:22.373405 11599 debug-util.cc:398] Leaking SignalData structure 0x7b080009b760 after lost signal to thread 11770
W20260504 19:59:22.374311 11599 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1f40 after lost signal to thread 11844
I20260504 19:59:22.378437 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:22.379004 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:23.380026 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:23.384981 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:23.385433 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:24.419992 11536 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20260504 19:59:24.425318 11820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:50508
I20260504 19:59:24.426215 11820 raft_consensus.cc:606] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:24.432729 11670 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55264
I20260504 19:59:24.433290 11670 raft_consensus.cc:606] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:24.435921 12011 raft_consensus.cc:993] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.436547 12011 raft_consensus.cc:1081] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.438493 11744 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55158
I20260504 19:59:24.439116 11744 raft_consensus.cc:493] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:24.439531 11744 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.443799 11670 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55264
I20260504 19:59:24.444566 11670 raft_consensus.cc:606] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:24.445680 11744 raft_consensus.cc:515] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.448972 11744 leader_election.cc:290] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 3 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:24.449705 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:24.449903 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:24.450608 11670 raft_consensus.cc:3055] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:24.450623 11820 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.451216 11670 raft_consensus.cc:740] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:24.452200 11669 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9928836addc24622ab5936846785460b"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55264
I20260504 19:59:24.452793 11669 raft_consensus.cc:606] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:24.452203 11670 consensus_queue.cc:260] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.453873 11670 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.458244 11820 raft_consensus.cc:2468] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 3.
I20260504 19:59:24.459527 11818 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
mode: GRACEFUL
new_leader_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:50508
I20260504 19:59:24.460109 11818 raft_consensus.cc:606] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 LEADER]: Received request to transfer leadership to TS 50b6b0a3f3224c01a95509c61e631d12
I20260504 19:59:24.463807 12011 raft_consensus.cc:993] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.464524 12011 raft_consensus.cc:1081] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.467008 11746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55158
I20260504 19:59:24.467585 11746 raft_consensus.cc:493] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:24.467944 11746 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.473218 11670 raft_consensus.cc:2468] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 3.
I20260504 19:59:24.477490 11746 raft_consensus.cc:515] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.478585 11709 leader_election.cc:304] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:24.482618 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:24.481654 11746 leader_election.cc:290] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 3 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:24.483515 11670 raft_consensus.cc:3055] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:24.484330 11670 raft_consensus.cc:740] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:24.485471 11670 consensus_queue.cc:260] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.486251 12012 raft_consensus.cc:2804] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:24.487064 11670 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.492901 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:24.493633 11820 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.495270 11670 raft_consensus.cc:2468] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 3.
I20260504 19:59:24.496707 11709 leader_election.cc:304] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:24.497830 12014 raft_consensus.cc:2804] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:24.497948 12012 raft_consensus.cc:697] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:24.498680 12014 raft_consensus.cc:697] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:24.499257 12012 consensus_queue.cc:237] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.499986 12014 consensus_queue.cc:237] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.502238 11820 raft_consensus.cc:2468] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 3.
I20260504 19:59:24.517582 11566 catalog_manager.cc:5671] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 2 to 3, leader changed from f65745628d11455ab10c4c37fc004db4 (127.11.68.1) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 3 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:24.519013 11566 catalog_manager.cc:5671] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 2 to 3, leader changed from f65745628d11455ab10c4c37fc004db4 (127.11.68.1) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 3 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:24.538352 12010 raft_consensus.cc:993] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.538865 12011 raft_consensus.cc:1081] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.540524 11746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9928836addc24622ab5936846785460b"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55158
I20260504 19:59:24.541098 11746 raft_consensus.cc:493] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:24.541404 11746 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.545387 11746 raft_consensus.cc:515] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.546922 11746 leader_election.cc:290] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 3 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:24.547770 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:24.548125 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:24.548414 11670 raft_consensus.cc:3055] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:24.548673 11820 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.548765 11670 raft_consensus.cc:740] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:24.549612 11670 consensus_queue.cc:260] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.550978 11670 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.555974 11820 raft_consensus.cc:2468] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 3.
I20260504 19:59:24.557410 11712 leader_election.cc:304] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:24.558075 11670 raft_consensus.cc:2468] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 3.
I20260504 19:59:24.558424 12014 raft_consensus.cc:2804] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:24.558992 12014 raft_consensus.cc:697] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:24.559963 12014 consensus_queue.cc:237] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.569499 11566 catalog_manager.cc:5671] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 2 to 3, leader changed from f65745628d11455ab10c4c37fc004db4 (127.11.68.1) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 3 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:24.641306 12013 raft_consensus.cc:993] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.641979 12008 raft_consensus.cc:1081] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.643703 11746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55146
I20260504 19:59:24.644212 11746 raft_consensus.cc:493] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:24.644526 11746 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.650810 11746 raft_consensus.cc:515] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:24.652649 11746 leader_election.cc:290] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 3 election: Requested vote from peers 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783), f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535)
I20260504 19:59:24.653999 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:24.654676 11670 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.654840 11820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:24.655616 11820 raft_consensus.cc:3055] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:24.656055 11820 raft_consensus.cc:740] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:24.657207 11820 consensus_queue.cc:260] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:24.659636 11820 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:24.663285 11670 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 3.
I20260504 19:59:24.664529 11709 leader_election.cc:304] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:24.665467 12014 raft_consensus.cc:2804] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:24.666154 12014 raft_consensus.cc:697] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:24.667510 12014 consensus_queue.cc:237] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:24.670204 11820 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 3.
I20260504 19:59:24.680490 11566 catalog_manager.cc:5671] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 2 to 3, leader changed from 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 3 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:24.965641 12013 raft_consensus.cc:993] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: : Instructing follower 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.966326 12008 raft_consensus.cc:1081] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 LEADER]: Signalling peer 50b6b0a3f3224c01a95509c61e631d12 to start an election
I20260504 19:59:24.968408 11746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
from {username='slave'} at 127.0.0.1:55146
I20260504 19:59:24.968984 11746 raft_consensus.cc:493] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:24.969282 11746 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:24.970023 11820 raft_consensus.cc:1275] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:24.972419 11819 raft_consensus.cc:1275] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:24.973258 12012 consensus_queue.cc:1048] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 19:59:24.976334 12012 consensus_queue.cc:1048] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:24.977216 11746 raft_consensus.cc:515] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.978787 11819 raft_consensus.cc:1275] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:24.980772 11746 leader_election.cc:290] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 4 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:24.981176 11670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:24.981366 12012 consensus_queue.cc:1048] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 19:59:24.982426 11670 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:24.986104 11818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916ddba7631c4852add1ea09e44294d5" candidate_uuid: "50b6b0a3f3224c01a95509c61e631d12" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:24.986809 11818 raft_consensus.cc:3055] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 19:59:24.987130 11818 raft_consensus.cc:740] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:24.988241 11818 consensus_queue.cc:260] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:24.989513 11818 raft_consensus.cc:3060] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:25.014369 11818 raft_consensus.cc:2468] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 4.
I20260504 19:59:25.018419 11712 leader_election.cc:304] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:25.037110 12014 raft_consensus.cc:2804] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 19:59:25.037773 12014 raft_consensus.cc:697] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 LEADER]: Becoming Leader. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:25.033416 11670 raft_consensus.cc:2468] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50b6b0a3f3224c01a95509c61e631d12 in term 4.
I20260504 19:59:25.050364 12014 consensus_queue.cc:237] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:25.055234 11670 raft_consensus.cc:1275] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:25.057142 11671 raft_consensus.cc:1275] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:25.058240 11668 raft_consensus.cc:1275] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:25.065111 12029 consensus_queue.cc:1048] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 19:59:25.070591 12029 consensus_queue.cc:1048] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 19:59:25.077270 12036 consensus_queue.cc:1048] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260504 19:59:25.087232 11566 catalog_manager.cc:5671] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 reported cstate change: term changed from 3 to 4, leader changed from 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3) to 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2). New cstate: current_term: 4 leader_uuid: "50b6b0a3f3224c01a95509c61e631d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:25.180537 11671 raft_consensus.cc:1275] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:25.182734 12029 consensus_queue.cc:1048] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 19:59:25.200021 11819 raft_consensus.cc:1275] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:25.202008 12029 consensus_queue.cc:1048] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 19:59:25.518265 11671 raft_consensus.cc:1275] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:25.519868 12029 consensus_queue.cc:1048] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 19:59:25.531024 11819 raft_consensus.cc:1275] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Refusing update from remote peer 50b6b0a3f3224c01a95509c61e631d12: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:25.533360 12036 consensus_queue.cc:1048] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 19:59:27.503710 11536 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20260504 19:59:27.504488 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:27.511907 11746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
mode: GRACEFUL
new_leader_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:38476
I20260504 19:59:27.512480 11746 raft_consensus.cc:606] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Received request to transfer leadership to TS f65745628d11455ab10c4c37fc004db4
I20260504 19:59:27.514840 11746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
mode: GRACEFUL
new_leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:38476
I20260504 19:59:27.515278 11746 raft_consensus.cc:606] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Received request to transfer leadership to TS 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:27.517365 11746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
mode: GRACEFUL
new_leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:38476
I20260504 19:59:27.517920 11746 raft_consensus.cc:606] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Received request to transfer leadership to TS 7b3ddc772c494eb4a7bf6bf1ec8ea600
I20260504 19:59:27.520042 11746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9928836addc24622ab5936846785460b"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
mode: GRACEFUL
new_leader_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:38476
I20260504 19:59:27.520475 11746 raft_consensus.cc:606] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Received request to transfer leadership to TS f65745628d11455ab10c4c37fc004db4
I20260504 19:59:27.522428 11746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb"
dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
mode: GRACEFUL
new_leader_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:38476
I20260504 19:59:27.522835 11746 raft_consensus.cc:606] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Received request to transfer leadership to TS f65745628d11455ab10c4c37fc004db4
I20260504 19:59:27.523836 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260504 19:59:27.524343 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:27.561077 12028 raft_consensus.cc:993] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12: : Instructing follower f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:27.561568 12052 raft_consensus.cc:1081] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Signalling peer f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:27.563197 11668 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:55236
I20260504 19:59:27.563731 11668 raft_consensus.cc:493] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:27.564042 11668 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:27.569428 11668 raft_consensus.cc:515] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.571689 11668 leader_election.cc:290] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 3 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:27.572686 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:27.573163 11816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7a5d6a6747434abed7c2460fe3d6fb" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:27.573357 11746 raft_consensus.cc:3055] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:27.573638 11746 raft_consensus.cc:740] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:27.573707 11816 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:27.574534 11746 consensus_queue.cc:260] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.575927 11746 raft_consensus.cc:3060] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:27.581257 11816 raft_consensus.cc:2468] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 3.
I20260504 19:59:27.582773 11637 leader_election.cc:304] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:27.582947 11746 raft_consensus.cc:2468] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 3.
I20260504 19:59:27.584466 12053 raft_consensus.cc:2804] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:27.585367 12053 raft_consensus.cc:697] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 3 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:27.586227 12053 consensus_queue.cc:237] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.592852 11566 catalog_manager.cc:5671] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 2 to 3, leader changed from 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2) to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:27.618350 12028 raft_consensus.cc:993] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12: : Instructing follower f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:27.618820 12052 raft_consensus.cc:1081] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Signalling peer f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:27.620218 11668 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9928836addc24622ab5936846785460b"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:55236
I20260504 19:59:27.620756 11668 raft_consensus.cc:493] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:27.621080 11668 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.623728 12028 raft_consensus.cc:993] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12: : Instructing follower 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:27.624403 12052 raft_consensus.cc:1081] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Signalling peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:27.626145 11816 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:50486
I20260504 19:59:27.626731 11816 raft_consensus.cc:493] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:27.627102 11816 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.627353 11668 raft_consensus.cc:515] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.629387 11668 leader_election.cc:290] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 4 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:27.630190 11746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:27.630414 11819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9928836addc24622ab5936846785460b" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:27.630748 11746 raft_consensus.cc:3055] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 19:59:27.630937 11819 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.631070 11746 raft_consensus.cc:740] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:27.631942 11746 consensus_queue.cc:260] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.632810 11746 raft_consensus.cc:3060] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.634958 11816 raft_consensus.cc:515] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:27.637872 11819 raft_consensus.cc:2468] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 4.
I20260504 19:59:27.638121 11816 leader_election.cc:290] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 4 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535)
I20260504 19:59:27.638756 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:27.639115 11637 leader_election.cc:304] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:27.639377 11744 raft_consensus.cc:3055] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 19:59:27.639740 11744 raft_consensus.cc:740] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:27.640407 12053 raft_consensus.cc:2804] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 19:59:27.640558 11668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d5d140270334e75b8e7d71189cced30" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:27.640394 11744 consensus_queue.cc:260] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:27.641156 12053 raft_consensus.cc:697] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 4 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:27.642105 11668 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.642228 11746 raft_consensus.cc:2468] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 4.
I20260504 19:59:27.642652 11744 raft_consensus.cc:3060] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.642962 12053 consensus_queue.cc:237] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.649741 11744 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 4.
I20260504 19:59:27.650831 11668 raft_consensus.cc:2468] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 4.
I20260504 19:59:27.650918 11785 leader_election.cc:304] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50b6b0a3f3224c01a95509c61e631d12, 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters:
I20260504 19:59:27.652259 12008 raft_consensus.cc:2804] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 19:59:27.653510 12008 raft_consensus.cc:697] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 LEADER]: Becoming Leader. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:27.653692 11566 catalog_manager.cc:5671] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 3 to 4, leader changed from 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2) to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 4 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:27.654525 12008 consensus_queue.cc:237] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } }
I20260504 19:59:27.662871 11567 catalog_manager.cc:5671] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 reported cstate change: term changed from 3 to 4, leader changed from 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2) to 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:27.697860 12028 raft_consensus.cc:993] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12: : Instructing follower f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:27.698385 12052 raft_consensus.cc:1081] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Signalling peer f65745628d11455ab10c4c37fc004db4 to start an election
I20260504 19:59:27.699970 11668 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4"
dest_uuid: "f65745628d11455ab10c4c37fc004db4"
from {username='slave'} at 127.0.0.1:55236
I20260504 19:59:27.700711 11668 raft_consensus.cc:493] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:27.701042 11668 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.706076 11668 raft_consensus.cc:515] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.708061 11668 leader_election.cc:290] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 4 election: Requested vote from peers 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859), 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3:34783)
I20260504 19:59:27.708961 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:27.709036 11816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08cff134f57545aea6a4360adb6b26f4" candidate_uuid: "f65745628d11455ab10c4c37fc004db4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
I20260504 19:59:27.709532 11744 raft_consensus.cc:3055] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 19:59:27.709636 11816 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.709954 11744 raft_consensus.cc:740] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:27.710811 11744 consensus_queue.cc:260] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.711742 11744 raft_consensus.cc:3060] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.715435 11816 raft_consensus.cc:2468] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 4.
I20260504 19:59:27.716487 11637 leader_election.cc:304] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:27.717154 12053 raft_consensus.cc:2804] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 19:59:27.717660 12053 raft_consensus.cc:697] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 4 LEADER]: Becoming Leader. State: Replica: f65745628d11455ab10c4c37fc004db4, State: Running, Role: LEADER
I20260504 19:59:27.718355 11744 raft_consensus.cc:2468] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f65745628d11455ab10c4c37fc004db4 in term 4.
I20260504 19:59:27.718698 12053 consensus_queue.cc:237] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.726272 11567 catalog_manager.cc:5671] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 reported cstate change: term changed from 3 to 4, leader changed from 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2) to f65745628d11455ab10c4c37fc004db4 (127.11.68.1). New cstate: current_term: 4 leader_uuid: "f65745628d11455ab10c4c37fc004db4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:27.880488 12028 raft_consensus.cc:993] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12: : Instructing follower 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:27.880945 12052 raft_consensus.cc:1081] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Signalling peer 7b3ddc772c494eb4a7bf6bf1ec8ea600 to start an election
I20260504 19:59:27.882488 11816 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e"
dest_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600"
from {username='slave'} at 127.0.0.1:50486
I20260504 19:59:27.882987 11816 raft_consensus.cc:493] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:27.883283 11816 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.888115 11816 raft_consensus.cc:515] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.889693 11816 leader_election.cc:290] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 4 election: Requested vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:27.890708 11744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "50b6b0a3f3224c01a95509c61e631d12"
I20260504 19:59:27.891191 11744 raft_consensus.cc:3055] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 19:59:27.890960 11668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3317460564e84b8a85fe4b6d6613223e" candidate_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f65745628d11455ab10c4c37fc004db4"
I20260504 19:59:27.891458 11744 raft_consensus.cc:740] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 50b6b0a3f3224c01a95509c61e631d12, State: Running, Role: LEADER
I20260504 19:59:27.891709 11668 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.892129 11744 consensus_queue.cc:260] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.892995 11744 raft_consensus.cc:3060] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:27.896757 11668 raft_consensus.cc:2468] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 4.
I20260504 19:59:27.897065 11744 raft_consensus.cc:2468] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b3ddc772c494eb4a7bf6bf1ec8ea600 in term 4.
I20260504 19:59:27.898198 11783 leader_election.cc:304] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600, f65745628d11455ab10c4c37fc004db4; no voters:
I20260504 19:59:27.898947 12008 raft_consensus.cc:2804] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 19:59:27.899434 12008 raft_consensus.cc:697] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 LEADER]: Becoming Leader. State: Replica: 7b3ddc772c494eb4a7bf6bf1ec8ea600, State: Running, Role: LEADER
I20260504 19:59:27.900148 12008 consensus_queue.cc:237] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:27.907927 11567 catalog_manager.cc:5671] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 reported cstate change: term changed from 3 to 4, leader changed from 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2) to 7b3ddc772c494eb4a7bf6bf1ec8ea600 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:28.044754 11816 raft_consensus.cc:1275] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:28.046092 12053 consensus_queue.cc:1048] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:28.056874 11744 raft_consensus.cc:1275] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:28.059814 12067 consensus_queue.cc:1048] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 19:59:28.069615 11816 raft_consensus.cc:1275] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:28.070951 12067 consensus_queue.cc:1048] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:28.081040 11744 raft_consensus.cc:1275] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:28.083499 12067 consensus_queue.cc:1048] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:28.200917 11668 raft_consensus.cc:1275] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:28.202350 12008 consensus_queue.cc:1048] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:28.211678 11744 raft_consensus.cc:1275] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:28.213593 12074 consensus_queue.cc:1048] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 19:59:28.250874 11744 raft_consensus.cc:1275] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:28.252090 12067 consensus_queue.cc:1048] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:28.261672 11816 raft_consensus.cc:1275] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Refusing update from remote peer f65745628d11455ab10c4c37fc004db4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:28.264003 12054 consensus_queue.cc:1048] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:28.397318 11668 raft_consensus.cc:1275] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:28.398774 12061 consensus_queue.cc:1048] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:28.408087 11746 raft_consensus.cc:1275] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Refusing update from remote peer 7b3ddc772c494eb4a7bf6bf1ec8ea600: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:28.409437 12061 consensus_queue.cc:1048] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:28.525586 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:28.530766 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:28.531220 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:29.532335 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:29.537371 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:29.537885 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:30.538995 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:30.543965 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:30.544458 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:31.545579 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:31.551945 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:31.552399 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:32.553436 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:32.558513 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:32.559003 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:33.560036 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:33.565403 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:33.565881 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:34.566964 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:34.572000 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:34.572491 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:35.667941 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:35.676362 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:35.676756 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260504 19:59:36.597527 11767 debug-util.cc:398] Leaking SignalData structure 0x7b0800127060 after lost signal to thread 11539
W20260504 19:59:36.598536 11767 debug-util.cc:398] Leaking SignalData structure 0x7b0800127740 after lost signal to thread 11602
W20260504 19:59:36.599524 11767 debug-util.cc:398] Leaking SignalData structure 0x7b08000ecec0 after lost signal to thread 11695
W20260504 19:59:36.600411 11767 debug-util.cc:398] Leaking SignalData structure 0x7b0800202720 after lost signal to thread 11770
W20260504 19:59:36.601330 11767 debug-util.cc:398] Leaking SignalData structure 0x7b08001a5be0 after lost signal to thread 11844
I20260504 19:59:36.678048 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:36.682893 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:36.683319 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:37.721524 11536 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20260504 19:59:37.722401 11536 tablet_server.cc:179] TabletServer@127.11.68.1:0 shutting down...
I20260504 19:59:37.763391 11536 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260504 19:59:37.764302 11536 tablet_replica.cc:333] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.765214 11536 raft_consensus.cc:2243] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 4 LEADER]: Raft consensus shutting down.
I20260504 19:59:37.766597 11536 raft_consensus.cc:2272] T 08cff134f57545aea6a4360adb6b26f4 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.769168 11536 tablet_replica.cc:333] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.769716 11536 raft_consensus.cc:2243] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:37.770277 11536 raft_consensus.cc:2272] T aa94336f31f248dfb81d211d4d4c8e3e P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.772121 11536 tablet_replica.cc:333] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.772696 11536 raft_consensus.cc:2243] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:37.773247 11536 raft_consensus.cc:2272] T 3d5d140270334e75b8e7d71189cced30 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.775141 11536 tablet_replica.cc:333] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.775647 11536 raft_consensus.cc:2243] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:37.776091 11536 raft_consensus.cc:2272] T decedeb8ba0a497ea6180094250bcc98 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.778002 11536 tablet_replica.cc:333] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.778530 11536 raft_consensus.cc:2243] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:37.779016 11536 raft_consensus.cc:2272] T 916ddba7631c4852add1ea09e44294d5 P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.780777 11536 tablet_replica.cc:333] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.781250 11536 raft_consensus.cc:2243] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:37.781702 11536 raft_consensus.cc:2272] T 3317460564e84b8a85fe4b6d6613223e P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.783525 11536 tablet_replica.cc:333] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.784014 11536 raft_consensus.cc:2243] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:37.784444 11536 raft_consensus.cc:2272] T ac981d4f70854ed184e117453b208344 P f65745628d11455ab10c4c37fc004db4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.786340 11536 tablet_replica.cc:333] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.786823 11536 raft_consensus.cc:2243] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 3 LEADER]: Raft consensus shutting down.
I20260504 19:59:37.787634 11536 raft_consensus.cc:2272] T df7a5d6a6747434abed7c2460fe3d6fb P f65745628d11455ab10c4c37fc004db4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.789963 11536 tablet_replica.cc:333] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4: stopping tablet replica
I20260504 19:59:37.790417 11536 raft_consensus.cc:2243] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 4 LEADER]: Raft consensus shutting down.
I20260504 19:59:37.791255 11536 raft_consensus.cc:2272] T 9928836addc24622ab5936846785460b P f65745628d11455ab10c4c37fc004db4 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260504 19:59:37.906651 11783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.68.1:34535: connect: Connection refused (error 111)
W20260504 19:59:37.911231 11783 consensus_peers.cc:597] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 -> Peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Couldn't send request to peer f65745628d11455ab10c4c37fc004db4. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.1:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 19:59:37.911484 11709 consensus_peers.cc:597] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 -> Peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Couldn't send request to peer f65745628d11455ab10c4c37fc004db4. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.1:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 19:59:37.917769 11536 tablet_server.cc:196] TabletServer@127.11.68.1:0 shutdown complete.
I20260504 19:59:37.949613 11536 tablet_server.cc:179] TabletServer@127.11.68.2:0 shutting down...
I20260504 19:59:37.989946 11536 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260504 19:59:37.990731 11536 tablet_replica.cc:333] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:37.991837 11536 raft_consensus.cc:2243] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:37.992493 11536 raft_consensus.cc:2272] T decedeb8ba0a497ea6180094250bcc98 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.994668 11536 tablet_replica.cc:333] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:37.995234 11536 raft_consensus.cc:2243] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:37.995699 11536 raft_consensus.cc:2272] T 3d5d140270334e75b8e7d71189cced30 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:37.997473 11536 tablet_replica.cc:333] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:37.998020 11536 raft_consensus.cc:2243] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Raft consensus shutting down.
I20260504 19:59:37.999025 11536 raft_consensus.cc:2272] T aa94336f31f248dfb81d211d4d4c8e3e P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.001051 11536 tablet_replica.cc:333] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:38.001556 11536 raft_consensus.cc:2243] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 LEADER]: Raft consensus shutting down.
I20260504 19:59:38.002481 11536 raft_consensus.cc:2272] T ac981d4f70854ed184e117453b208344 P 50b6b0a3f3224c01a95509c61e631d12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.004441 11536 tablet_replica.cc:333] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:38.005018 11536 raft_consensus.cc:2243] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.005560 11536 raft_consensus.cc:2272] T 3317460564e84b8a85fe4b6d6613223e P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.007506 11536 tablet_replica.cc:333] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:38.008256 11536 raft_consensus.cc:2243] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 LEADER]: Raft consensus shutting down.
I20260504 19:59:38.009116 11536 raft_consensus.cc:2272] T 916ddba7631c4852add1ea09e44294d5 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.011053 11536 tablet_replica.cc:333] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:38.011545 11536 raft_consensus.cc:2243] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.012035 11536 raft_consensus.cc:2272] T df7a5d6a6747434abed7c2460fe3d6fb P 50b6b0a3f3224c01a95509c61e631d12 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.014015 11536 tablet_replica.cc:333] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:38.014497 11536 raft_consensus.cc:2243] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.014950 11536 raft_consensus.cc:2272] T 9928836addc24622ab5936846785460b P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.016677 11536 tablet_replica.cc:333] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12: stopping tablet replica
I20260504 19:59:38.017127 11536 raft_consensus.cc:2243] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.017611 11536 raft_consensus.cc:2272] T 08cff134f57545aea6a4360adb6b26f4 P 50b6b0a3f3224c01a95509c61e631d12 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260504 19:59:38.209931 11785 consensus_peers.cc:597] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 -> Peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Couldn't send request to peer 50b6b0a3f3224c01a95509c61e631d12. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.2:37859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 19:59:38.236816 11785 consensus_peers.cc:597] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 -> Peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Couldn't send request to peer 50b6b0a3f3224c01a95509c61e631d12. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.2:37859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 19:59:38.237627 11783 consensus_peers.cc:597] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 -> Peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Couldn't send request to peer f65745628d11455ab10c4c37fc004db4. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.1:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 19:59:38.265511 11783 consensus_peers.cc:597] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 -> Peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Couldn't send request to peer f65745628d11455ab10c4c37fc004db4. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.1:34535: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 19:59:38.337262 11785 consensus_peers.cc:597] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 -> Peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Couldn't send request to peer 50b6b0a3f3224c01a95509c61e631d12. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.2:37859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 19:59:38.887183 12096 raft_consensus.cc:493] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f65745628d11455ab10c4c37fc004db4)
I20260504 19:59:38.888137 12096 raft_consensus.cc:515] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
I20260504 19:59:38.890719 12096 leader_election.cc:290] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
I20260504 19:59:38.894162 12096 raft_consensus.cc:493] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f65745628d11455ab10c4c37fc004db4)
W20260504 19:59:38.894296 11783 leader_election.cc:336] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Network error: Client connection negotiation failed: client connection to 127.11.68.1:34535: connect: Connection refused (error 111)
I20260504 19:59:38.894701 12096 raft_consensus.cc:515] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65745628d11455ab10c4c37fc004db4" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 34535 } } peers { permanent_uuid: "50b6b0a3f3224c01a95509c61e631d12" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 37859 } } peers { permanent_uuid: "7b3ddc772c494eb4a7bf6bf1ec8ea600" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 34783 } }
W20260504 19:59:38.895242 11785 leader_election.cc:336] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Network error: Client connection negotiation failed: client connection to 127.11.68.2:37859: connect: Connection refused (error 111)
I20260504 19:59:38.895673 11785 leader_election.cc:304] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4
I20260504 19:59:38.896307 12094 raft_consensus.cc:2749] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 19:59:38.896662 12096 leader_election.cc:290] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535), 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859)
W20260504 19:59:38.900758 11783 leader_election.cc:336] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f65745628d11455ab10c4c37fc004db4 (127.11.68.1:34535): Network error: Client connection negotiation failed: client connection to 127.11.68.1:34535: connect: Connection refused (error 111)
W20260504 19:59:38.901304 11785 leader_election.cc:336] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 50b6b0a3f3224c01a95509c61e631d12 (127.11.68.2:37859): Network error: Client connection negotiation failed: client connection to 127.11.68.2:37859: connect: Connection refused (error 111)
I20260504 19:59:38.901608 11785 leader_election.cc:304] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b3ddc772c494eb4a7bf6bf1ec8ea600; no voters: 50b6b0a3f3224c01a95509c61e631d12, f65745628d11455ab10c4c37fc004db4
I20260504 19:59:38.902199 12096 raft_consensus.cc:2749] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 19:59:38.911650 11536 tablet_server.cc:196] TabletServer@127.11.68.2:0 shutdown complete.
I20260504 19:59:38.932168 11536 tablet_server.cc:179] TabletServer@127.11.68.3:0 shutting down...
I20260504 19:59:38.958000 11536 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260504 19:59:38.958846 11536 tablet_replica.cc:333] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.959872 11536 raft_consensus.cc:2243] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.960598 11536 raft_consensus.cc:2272] T df7a5d6a6747434abed7c2460fe3d6fb P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.962630 11536 tablet_replica.cc:333] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.963127 11536 raft_consensus.cc:2243] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 LEADER]: Raft consensus shutting down.
I20260504 19:59:38.963802 11536 raft_consensus.cc:2272] T decedeb8ba0a497ea6180094250bcc98 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.965479 11536 tablet_replica.cc:333] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.966040 11536 raft_consensus.cc:2243] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.966482 11536 raft_consensus.cc:2272] T 9928836addc24622ab5936846785460b P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.968128 11536 tablet_replica.cc:333] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.968593 11536 raft_consensus.cc:2243] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.968963 11536 raft_consensus.cc:2272] T 916ddba7631c4852add1ea09e44294d5 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.970654 11536 tablet_replica.cc:333] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.971069 11536 raft_consensus.cc:2243] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.971483 11536 raft_consensus.cc:2272] T ac981d4f70854ed184e117453b208344 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.973479 11536 tablet_replica.cc:333] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.974128 11536 raft_consensus.cc:2243] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 LEADER]: Raft consensus shutting down.
I20260504 19:59:38.974886 11536 raft_consensus.cc:2272] T 3317460564e84b8a85fe4b6d6613223e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.977176 11536 tablet_replica.cc:333] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.977658 11536 raft_consensus.cc:2243] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.978139 11536 raft_consensus.cc:2272] T aa94336f31f248dfb81d211d4d4c8e3e P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.979839 11536 tablet_replica.cc:333] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.980297 11536 raft_consensus.cc:2243] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 LEADER]: Raft consensus shutting down.
I20260504 19:59:38.981006 11536 raft_consensus.cc:2272] T 3d5d140270334e75b8e7d71189cced30 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:38.982831 11536 tablet_replica.cc:333] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600: stopping tablet replica
I20260504 19:59:38.983286 11536 raft_consensus.cc:2243] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 19:59:38.983727 11536 raft_consensus.cc:2272] T 08cff134f57545aea6a4360adb6b26f4 P 7b3ddc772c494eb4a7bf6bf1ec8ea600 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:39.013751 11536 tablet_server.cc:196] TabletServer@127.11.68.3:0 shutdown complete.
I20260504 19:59:39.032838 11536 master.cc:562] Master@127.11.68.62:40525 shutting down...
I20260504 19:59:39.147030 11536 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 1 LEADER]: Raft consensus shutting down.
I20260504 19:59:39.147740 11536 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 19:59:39.148130 11536 tablet_replica.cc:333] T 00000000000000000000000000000000 P 42967374b8734f19850eb44b7a55611c: stopping tablet replica
I20260504 19:59:39.187536 11536 master.cc:584] Master@127.11.68.62:40525 shutdown complete.
50b6b0a3f3224c01a95509c61e631d12 4
7b3ddc772c494eb4a7bf6bf1ec8ea600 1
f65745628d11455ab10c4c37fc004db4 4
50b6b0a3f3224c01a95509c61e631d12 3
7b3ddc772c494eb4a7bf6bf1ec8ea600 3
f65745628d11455ab10c4c37fc004db4 3
50b6b0a3f3224c01a95509c61e631d12 8
7b3ddc772c494eb4a7bf6bf1ec8ea600 1
f65745628d11455ab10c4c37fc004db4 0
50b6b0a3f3224c01a95509c61e631d12 3
7b3ddc772c494eb4a7bf6bf1ec8ea600 3
f65745628d11455ab10c4c37fc004db4 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (34024 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260504 19:59:39.234385 11536 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.68.62:37821
I20260504 19:59:39.235632 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 19:59:39.241677 12099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:39.241781 12098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:39.242441 12101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:39.244474 11536 server_base.cc:1061] running on GCE node
I20260504 19:59:39.245438 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 19:59:39.245626 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 19:59:39.245750 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924779245739 us; error 0 us; skew 500 ppm
I20260504 19:59:39.246367 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 19:59:39.248857 11536 webserver.cc:492] Webserver started at http://127.11.68.62:35669/ using document root <none> and password file <none>
I20260504 19:59:39.249437 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 19:59:39.249637 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 19:59:39.249955 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 19:59:39.251192 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/master-0-root/instance:
uuid: "ca7ec6b6819b4afb9fde3ea796094e2d"
format_stamp: "Formatted at 2026-05-04 19:59:39 on dist-test-slave-3x8x"
I20260504 19:59:39.255889 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260504 19:59:39.259460 12106 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.260365 11536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260504 19:59:39.260640 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/master-0-root
uuid: "ca7ec6b6819b4afb9fde3ea796094e2d"
format_stamp: "Formatted at 2026-05-04 19:59:39 on dist-test-slave-3x8x"
I20260504 19:59:39.260890 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 19:59:39.275720 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 19:59:39.276975 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 19:59:39.315881 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.62:37821
I20260504 19:59:39.315969 12157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.62:37821 every 8 connection(s)
I20260504 19:59:39.319996 12158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.329485 12158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d: Bootstrap starting.
I20260504 19:59:39.334412 12158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:39.338783 12158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d: No bootstrap required, opened a new log
I20260504 19:59:39.341176 12158 raft_consensus.cc:359] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7ec6b6819b4afb9fde3ea796094e2d" member_type: VOTER }
I20260504 19:59:39.341576 12158 raft_consensus.cc:385] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:39.341905 12158 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7ec6b6819b4afb9fde3ea796094e2d, State: Initialized, Role: FOLLOWER
I20260504 19:59:39.342396 12158 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7ec6b6819b4afb9fde3ea796094e2d" member_type: VOTER }
I20260504 19:59:39.342821 12158 raft_consensus.cc:399] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260504 19:59:39.343010 12158 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260504 19:59:39.343221 12158 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:39.347282 12158 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7ec6b6819b4afb9fde3ea796094e2d" member_type: VOTER }
I20260504 19:59:39.347785 12158 leader_election.cc:304] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7ec6b6819b4afb9fde3ea796094e2d; no voters:
I20260504 19:59:39.348868 12158 leader_election.cc:290] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260504 19:59:39.349274 12161 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:39.350688 12161 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 1 LEADER]: Becoming Leader. State: Replica: ca7ec6b6819b4afb9fde3ea796094e2d, State: Running, Role: LEADER
I20260504 19:59:39.351482 12161 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7ec6b6819b4afb9fde3ea796094e2d" member_type: VOTER }
I20260504 19:59:39.352088 12158 sys_catalog.cc:565] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [sys.catalog]: configured and running, proceeding with master startup.
I20260504 19:59:39.356360 12163 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7ec6b6819b4afb9fde3ea796094e2d. Latest consensus state: current_term: 1 leader_uuid: "ca7ec6b6819b4afb9fde3ea796094e2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7ec6b6819b4afb9fde3ea796094e2d" member_type: VOTER } }
I20260504 19:59:39.357004 12163 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [sys.catalog]: This master's current role is: LEADER
I20260504 19:59:39.358412 12162 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7ec6b6819b4afb9fde3ea796094e2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7ec6b6819b4afb9fde3ea796094e2d" member_type: VOTER } }
I20260504 19:59:39.360492 12162 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [sys.catalog]: This master's current role is: LEADER
I20260504 19:59:39.361115 12170 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 19:59:39.367847 12170 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 19:59:39.368860 11536 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260504 19:59:39.376857 12170 catalog_manager.cc:1357] Generated new cluster ID: 4422631753a14dc888db20c8afdfd5d1
I20260504 19:59:39.377195 12170 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 19:59:39.404783 12170 catalog_manager.cc:1380] Generated new certificate authority record
I20260504 19:59:39.406117 12170 catalog_manager.cc:1514] Loading token signing keys...
I20260504 19:59:39.430145 12170 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d: Generated new TSK 0
I20260504 19:59:39.430796 12170 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 19:59:39.435839 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 19:59:39.442420 12179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:39.442739 12180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:39.445466 12182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:39.446053 11536 server_base.cc:1061] running on GCE node
I20260504 19:59:39.446969 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 19:59:39.447197 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 19:59:39.447398 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924779447376 us; error 0 us; skew 500 ppm
I20260504 19:59:39.447922 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 19:59:39.450557 11536 webserver.cc:492] Webserver started at http://127.11.68.1:42465/ using document root <none> and password file <none>
I20260504 19:59:39.451093 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 19:59:39.451309 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 19:59:39.451599 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 19:59:39.452701 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-0-root/instance:
uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
format_stamp: "Formatted at 2026-05-04 19:59:39 on dist-test-slave-3x8x"
I20260504 19:59:39.457399 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260504 19:59:39.461462 12187 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.462368 11536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260504 19:59:39.462756 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-0-root
uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
format_stamp: "Formatted at 2026-05-04 19:59:39 on dist-test-slave-3x8x"
I20260504 19:59:39.463071 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 19:59:39.481592 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 19:59:39.482988 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 19:59:39.484630 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 19:59:39.487465 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 19:59:39.487715 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.487985 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 19:59:39.488171 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.530750 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.1:35487
I20260504 19:59:39.530841 12249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.1:35487 every 8 connection(s)
I20260504 19:59:39.535451 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 19:59:39.543848 12254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:39.544930 12255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:39.547776 11536 server_base.cc:1061] running on GCE node
W20260504 19:59:39.548242 12257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:39.549193 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 19:59:39.549414 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 19:59:39.549582 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924779549566 us; error 0 us; skew 500 ppm
I20260504 19:59:39.550177 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 19:59:39.553331 11536 webserver.cc:492] Webserver started at http://127.11.68.2:40105/ using document root <none> and password file <none>
I20260504 19:59:39.554105 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 19:59:39.554041 12250 heartbeater.cc:344] Connected to a master server at 127.11.68.62:37821
I20260504 19:59:39.554503 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 19:59:39.554623 12250 heartbeater.cc:461] Registering TS with master...
I20260504 19:59:39.554884 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 19:59:39.555572 12250 heartbeater.cc:507] Master 127.11.68.62:37821 requested a full tablet report, sending...
I20260504 19:59:39.556387 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-1-root/instance:
uuid: "f9a4e75173b54be181f23568394a3a8d"
format_stamp: "Formatted at 2026-05-04 19:59:39 on dist-test-slave-3x8x"
I20260504 19:59:39.557984 12123 ts_manager.cc:194] Registered new tserver with Master: 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487)
I20260504 19:59:39.559775 12123 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54198
I20260504 19:59:39.561667 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260504 19:59:39.565753 12262 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.566656 11536 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260504 19:59:39.566906 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-1-root
uuid: "f9a4e75173b54be181f23568394a3a8d"
format_stamp: "Formatted at 2026-05-04 19:59:39 on dist-test-slave-3x8x"
I20260504 19:59:39.567149 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 19:59:39.591928 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 19:59:39.593120 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 19:59:39.594619 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 19:59:39.597227 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 19:59:39.597518 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.597795 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 19:59:39.598001 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260504 19:59:39.639796 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.2:45537
I20260504 19:59:39.639919 12324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.2:45537 every 8 connection(s)
I20260504 19:59:39.644801 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 19:59:39.651882 12328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 19:59:39.654141 12329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:39.656867 11536 server_base.cc:1061] running on GCE node
W20260504 19:59:39.656978 12331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 19:59:39.657361 12325 heartbeater.cc:344] Connected to a master server at 127.11.68.62:37821
I20260504 19:59:39.657877 12325 heartbeater.cc:461] Registering TS with master...
I20260504 19:59:39.658187 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 19:59:39.658427 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 19:59:39.658598 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924779658583 us; error 0 us; skew 500 ppm
I20260504 19:59:39.658700 12325 heartbeater.cc:507] Master 127.11.68.62:37821 requested a full tablet report, sending...
I20260504 19:59:39.659190 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 19:59:39.660972 12123 ts_manager.cc:194] Registered new tserver with Master: f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:39.662132 11536 webserver.cc:492] Webserver started at http://127.11.68.3:42355/ using document root <none> and password file <none>
I20260504 19:59:39.662846 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 19:59:39.662915 12123 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54206
I20260504 19:59:39.663149 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 19:59:39.663488 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 19:59:39.665150 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-2-root/instance:
uuid: "bc677a5b14684d43b945885131fe0922"
format_stamp: "Formatted at 2026-05-04 19:59:39 on dist-test-slave-3x8x"
I20260504 19:59:39.670301 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260504 19:59:39.673758 12336 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.674690 11536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260504 19:59:39.675019 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-2-root
uuid: "bc677a5b14684d43b945885131fe0922"
format_stamp: "Formatted at 2026-05-04 19:59:39 on dist-test-slave-3x8x"
I20260504 19:59:39.675282 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777924745039007-11536-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 19:59:39.683678 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 19:59:39.685379 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 19:59:39.687512 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 19:59:39.689924 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 19:59:39.690119 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.690306 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 19:59:39.690431 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 19:59:39.732385 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.3:38647
I20260504 19:59:39.732498 12398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.3:38647 every 8 connection(s)
I20260504 19:59:39.746773 12399 heartbeater.cc:344] Connected to a master server at 127.11.68.62:37821
I20260504 19:59:39.747193 12399 heartbeater.cc:461] Registering TS with master...
I20260504 19:59:39.748054 12399 heartbeater.cc:507] Master 127.11.68.62:37821 requested a full tablet report, sending...
I20260504 19:59:39.750288 12123 ts_manager.cc:194] Registered new tserver with Master: bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:39.750401 11536 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014889854s
I20260504 19:59:39.751780 12123 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54220
I20260504 19:59:39.776448 12123 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54232:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 19:59:39.779474 12123 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 19:59:39.853147 12364 tablet_service.cc:1511] Processing CreateTablet for tablet 24d5d169f54c4b198069cf44a83e1289 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260504 19:59:39.853701 12362 tablet_service.cc:1511] Processing CreateTablet for tablet a2fd6cd6b68b47f999acb5ea52fde71e (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260504 19:59:39.854883 12364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d5d169f54c4b198069cf44a83e1289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.856115 12363 tablet_service.cc:1511] Processing CreateTablet for tablet 0bbfa5c44a6d4d61a6175fe1e4a62d53 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260504 19:59:39.856212 12361 tablet_service.cc:1511] Processing CreateTablet for tablet d4746e573b2142a09b3bc863b77051bd (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260504 19:59:39.857690 12363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bbfa5c44a6d4d61a6175fe1e4a62d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.858409 12361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4746e573b2142a09b3bc863b77051bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.859118 12362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fd6cd6b68b47f999acb5ea52fde71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.870260 12360 tablet_service.cc:1511] Processing CreateTablet for tablet cdaee4dfea3548dcb8579e74423edb9c (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260504 19:59:39.872272 12360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdaee4dfea3548dcb8579e74423edb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.876837 12359 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd1d29b2cc94b5390cc815b5287f426 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260504 19:59:39.878314 12359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd1d29b2cc94b5390cc815b5287f426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.883824 12358 tablet_service.cc:1511] Processing CreateTablet for tablet 734e7619181643f7a33111df773337b0 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260504 19:59:39.885878 12358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734e7619181643f7a33111df773337b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.892328 12363 tablet_service.cc:1511] Processing CreateTablet for tablet 55a11f407d464fbe8753eed9f00ef952 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260504 19:59:39.893853 12363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a11f407d464fbe8753eed9f00ef952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.897406 12357 tablet_service.cc:1511] Processing CreateTablet for tablet 06a32550992a4221907f6d6a1bd147dd (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260504 19:59:39.899001 12357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06a32550992a4221907f6d6a1bd147dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.900846 12356 tablet_service.cc:1511] Processing CreateTablet for tablet 27d2807b93ae4423aeeeae7b5334de70 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260504 19:59:39.902274 12356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d2807b93ae4423aeeeae7b5334de70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.909212 12215 tablet_service.cc:1511] Processing CreateTablet for tablet 24d5d169f54c4b198069cf44a83e1289 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260504 19:59:39.910825 12215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d5d169f54c4b198069cf44a83e1289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.915550 12213 tablet_service.cc:1511] Processing CreateTablet for tablet a2fd6cd6b68b47f999acb5ea52fde71e (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260504 19:59:39.917562 12213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fd6cd6b68b47f999acb5ea52fde71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.913586 12214 tablet_service.cc:1511] Processing CreateTablet for tablet 0bbfa5c44a6d4d61a6175fe1e4a62d53 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260504 19:59:39.920266 12214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bbfa5c44a6d4d61a6175fe1e4a62d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.925721 12212 tablet_service.cc:1511] Processing CreateTablet for tablet d4746e573b2142a09b3bc863b77051bd (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260504 19:59:39.927192 12212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4746e573b2142a09b3bc863b77051bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.935747 12211 tablet_service.cc:1511] Processing CreateTablet for tablet cdaee4dfea3548dcb8579e74423edb9c (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260504 19:59:39.937211 12211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdaee4dfea3548dcb8579e74423edb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.939103 12210 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd1d29b2cc94b5390cc815b5287f426 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260504 19:59:39.939761 12284 tablet_service.cc:1511] Processing CreateTablet for tablet 734e7619181643f7a33111df773337b0 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260504 19:59:39.940536 12210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd1d29b2cc94b5390cc815b5287f426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.941962 12284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734e7619181643f7a33111df773337b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.943029 12209 tablet_service.cc:1511] Processing CreateTablet for tablet 734e7619181643f7a33111df773337b0 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260504 19:59:39.944379 12209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734e7619181643f7a33111df773337b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.947309 12290 tablet_service.cc:1511] Processing CreateTablet for tablet 24d5d169f54c4b198069cf44a83e1289 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260504 19:59:39.947551 12287 tablet_service.cc:1511] Processing CreateTablet for tablet d4746e573b2142a09b3bc863b77051bd (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260504 19:59:39.947758 12288 tablet_service.cc:1511] Processing CreateTablet for tablet a2fd6cd6b68b47f999acb5ea52fde71e (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260504 19:59:39.948772 12290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24d5d169f54c4b198069cf44a83e1289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.949787 12206 tablet_service.cc:1511] Processing CreateTablet for tablet 27d2807b93ae4423aeeeae7b5334de70 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260504 19:59:39.950222 12285 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd1d29b2cc94b5390cc815b5287f426 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260504 19:59:39.951378 12206 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d2807b93ae4423aeeeae7b5334de70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.951051 12208 tablet_service.cc:1511] Processing CreateTablet for tablet 55a11f407d464fbe8753eed9f00ef952 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260504 19:59:39.952582 12208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a11f407d464fbe8753eed9f00ef952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.952190 12286 tablet_service.cc:1511] Processing CreateTablet for tablet cdaee4dfea3548dcb8579e74423edb9c (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260504 19:59:39.954596 12289 tablet_service.cc:1511] Processing CreateTablet for tablet 0bbfa5c44a6d4d61a6175fe1e4a62d53 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260504 19:59:39.956610 12283 tablet_service.cc:1511] Processing CreateTablet for tablet 55a11f407d464fbe8753eed9f00ef952 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260504 19:59:39.956857 12207 tablet_service.cc:1511] Processing CreateTablet for tablet 06a32550992a4221907f6d6a1bd147dd (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260504 19:59:39.958257 12207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06a32550992a4221907f6d6a1bd147dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.959020 12282 tablet_service.cc:1511] Processing CreateTablet for tablet 06a32550992a4221907f6d6a1bd147dd (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260504 19:59:39.963470 12287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4746e573b2142a09b3bc863b77051bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.964267 12281 tablet_service.cc:1511] Processing CreateTablet for tablet 27d2807b93ae4423aeeeae7b5334de70 (DEFAULT_TABLE table=test-workload [id=f514627fdfcc41af8bc62b913f17b9ce]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260504 19:59:39.964824 12288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fd6cd6b68b47f999acb5ea52fde71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.966116 12289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bbfa5c44a6d4d61a6175fe1e4a62d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.969090 12283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55a11f407d464fbe8753eed9f00ef952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.969745 12282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06a32550992a4221907f6d6a1bd147dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.972101 12281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27d2807b93ae4423aeeeae7b5334de70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.981050 12285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd1d29b2cc94b5390cc815b5287f426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.982249 12286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdaee4dfea3548dcb8579e74423edb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 19:59:39.998708 12413 tablet_bootstrap.cc:492] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.014086 12414 tablet_bootstrap.cc:492] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.034219 12413 tablet_bootstrap.cc:654] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.040339 12415 tablet_bootstrap.cc:492] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.067480 12413 tablet_bootstrap.cc:492] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.067727 12414 tablet_bootstrap.cc:654] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.068020 12413 ts_tablet_manager.cc:1403] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.070s user 0.027s sys 0.007s
I20260504 19:59:40.071069 12413 raft_consensus.cc:359] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.071847 12413 raft_consensus.cc:385] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.072153 12413 raft_consensus.cc:740] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.072867 12413 consensus_queue.cc:260] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.092893 12415 tablet_bootstrap.cc:654] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.096251 12414 tablet_bootstrap.cc:492] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.096938 12414 ts_tablet_manager.cc:1403] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.083s user 0.021s sys 0.013s
I20260504 19:59:40.097662 12399 heartbeater.cc:499] Master 127.11.68.62:37821 was elected leader, sending a full tablet report...
I20260504 19:59:40.098294 12413 ts_tablet_manager.cc:1434] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.030s user 0.018s sys 0.005s
I20260504 19:59:40.099296 12413 tablet_bootstrap.cc:492] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.100112 12414 raft_consensus.cc:359] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.100921 12414 raft_consensus.cc:385] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.101233 12414 raft_consensus.cc:740] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.102090 12414 consensus_queue.cc:260] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.106004 12413 tablet_bootstrap.cc:654] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.111570 12415 tablet_bootstrap.cc:492] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.111987 12414 ts_tablet_manager.cc:1434] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260504 19:59:40.112164 12415 ts_tablet_manager.cc:1403] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.015s
I20260504 19:59:40.112581 12250 heartbeater.cc:499] Master 127.11.68.62:37821 was elected leader, sending a full tablet report...
I20260504 19:59:40.115319 12415 raft_consensus.cc:359] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.116434 12415 raft_consensus.cc:385] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.116793 12415 raft_consensus.cc:740] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.117763 12415 consensus_queue.cc:260] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.118877 12414 tablet_bootstrap.cc:492] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.127463 12415 ts_tablet_manager.cc:1434] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
I20260504 19:59:40.127795 12414 tablet_bootstrap.cc:654] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.128444 12325 heartbeater.cc:499] Master 127.11.68.62:37821 was elected leader, sending a full tablet report...
I20260504 19:59:40.139412 12415 tablet_bootstrap.cc:492] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.141448 12413 tablet_bootstrap.cc:492] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.142007 12413 ts_tablet_manager.cc:1403] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.043s user 0.018s sys 0.018s
W20260504 19:59:40.144385 12326 tablet.cc:2404] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 19:59:40.145125 12413 raft_consensus.cc:359] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.146106 12413 raft_consensus.cc:385] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.146400 12413 raft_consensus.cc:740] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.147187 12413 consensus_queue.cc:260] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.149408 12413 ts_tablet_manager.cc:1434] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260504 19:59:40.150403 12413 tablet_bootstrap.cc:492] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.151445 12418 raft_consensus.cc:493] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.151962 12418 raft_consensus.cc:515] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.156694 12413 tablet_bootstrap.cc:654] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.158505 12418 leader_election.cc:290] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.197441 12415 tablet_bootstrap.cc:654] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.198489 12413 tablet_bootstrap.cc:492] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.199054 12413 ts_tablet_manager.cc:1403] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.049s user 0.029s sys 0.017s
I20260504 19:59:40.201911 12414 tablet_bootstrap.cc:492] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.202451 12414 ts_tablet_manager.cc:1403] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.084s user 0.036s sys 0.033s
I20260504 19:59:40.202307 12413 raft_consensus.cc:359] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.203073 12413 raft_consensus.cc:385] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.203379 12413 raft_consensus.cc:740] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.205062 12413 consensus_queue.cc:260] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.208323 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bbfa5c44a6d4d61a6175fe1e4a62d53" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.208843 12413 ts_tablet_manager.cc:1434] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20260504 19:59:40.209779 12413 tablet_bootstrap.cc:492] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.205574 12414 raft_consensus.cc:359] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.214025 12414 raft_consensus.cc:385] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.206548 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bbfa5c44a6d4d61a6175fe1e4a62d53" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.214397 12414 raft_consensus.cc:740] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
W20260504 19:59:40.215446 12340 leader_election.cc:343] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.215128 12414 consensus_queue.cc:260] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.216037 12413 tablet_bootstrap.cc:654] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.216248 12225 raft_consensus.cc:2468] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 0.
I20260504 19:59:40.207105 12415 tablet_bootstrap.cc:492] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.217406 12415 ts_tablet_manager.cc:1403] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.078s user 0.013s sys 0.001s
I20260504 19:59:40.220530 12415 raft_consensus.cc:359] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.221230 12415 raft_consensus.cc:385] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.221522 12415 raft_consensus.cc:740] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.221954 12414 ts_tablet_manager.cc:1434] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260504 19:59:40.222332 12415 consensus_queue.cc:260] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.223204 12414 tablet_bootstrap.cc:492] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.225747 12422 raft_consensus.cc:493] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.226223 12422 raft_consensus.cc:515] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.227242 12340 leader_election.cc:304] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters: f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:40.240552 12418 raft_consensus.cc:2804] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.241003 12418 raft_consensus.cc:493] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.241406 12418 raft_consensus.cc:3060] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.242985 12414 tablet_bootstrap.cc:654] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.245850 12413 tablet_bootstrap.cc:492] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.246608 12413 ts_tablet_manager.cc:1403] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.020s
I20260504 19:59:40.250674 12413 raft_consensus.cc:359] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.251456 12413 raft_consensus.cc:385] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.251791 12413 raft_consensus.cc:740] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.252597 12413 consensus_queue.cc:260] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.253270 12414 tablet_bootstrap.cc:492] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.228864 12422 leader_election.cc:290] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.253953 12414 ts_tablet_manager.cc:1403] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.008s
I20260504 19:59:40.254194 12422 raft_consensus.cc:493] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.254813 12422 raft_consensus.cc:515] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.257203 12422 leader_election.cc:290] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.258004 12414 raft_consensus.cc:359] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.259012 12414 raft_consensus.cc:385] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.259414 12414 raft_consensus.cc:740] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.260532 12414 consensus_queue.cc:260] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.263005 12413 ts_tablet_manager.cc:1434] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260504 19:59:40.263633 12415 ts_tablet_manager.cc:1434] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20260504 19:59:40.264654 12415 tablet_bootstrap.cc:492] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.265501 12414 ts_tablet_manager.cc:1434] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260504 19:59:40.266536 12414 tablet_bootstrap.cc:492] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.266228 12418 raft_consensus.cc:515] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.269008 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bbfa5c44a6d4d61a6175fe1e4a62d53" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:40.269646 12225 raft_consensus.cc:3060] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.271634 12413 tablet_bootstrap.cc:492] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.273264 12414 tablet_bootstrap.cc:654] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.277609 12225 raft_consensus.cc:2468] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 1.
I20260504 19:59:40.278680 12432 raft_consensus.cc:493] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.278935 12340 leader_election.cc:304] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.280798 12420 raft_consensus.cc:493] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.279511 12432 raft_consensus.cc:515] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.281467 12420 raft_consensus.cc:515] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.284852 12414 tablet_bootstrap.cc:492] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.285387 12414 ts_tablet_manager.cc:1403] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20260504 19:59:40.288436 12420 leader_election.cc:290] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.288518 12414 raft_consensus.cc:359] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.292162 12414 raft_consensus.cc:385] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.292577 12414 raft_consensus.cc:740] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.292977 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bbfa5c44a6d4d61a6175fe1e4a62d53" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:40.293330 12414 consensus_queue.cc:260] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
W20260504 19:59:40.294346 12340 leader_election.cc:343] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.294879 12418 leader_election.cc:290] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.298631 12414 ts_tablet_manager.cc:1434] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s
I20260504 19:59:40.300006 12414 tablet_bootstrap.cc:492] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.290043 12434 raft_consensus.cc:2804] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:40.308960 12434 raft_consensus.cc:697] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:40.309311 12414 tablet_bootstrap.cc:654] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.310276 12434 consensus_queue.cc:237] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.315915 12415 tablet_bootstrap.cc:654] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.317961 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.318032 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06a32550992a4221907f6d6a1bd147dd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.319659 12224 raft_consensus.cc:2468] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 0.
W20260504 19:59:40.320734 12266 leader_election.cc:343] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.323596 12266 leader_election.cc:304] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:40.324796 12431 raft_consensus.cc:2804] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.325313 12431 raft_consensus.cc:493] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.325716 12431 raft_consensus.cc:3060] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.334408 12374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:40.336334 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06a32550992a4221907f6d6a1bd147dd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:40.338069 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06a32550992a4221907f6d6a1bd147dd" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
W20260504 19:59:40.338866 12264 leader_election.cc:343] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.339339 12264 leader_election.cc:304] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a4e75173b54be181f23568394a3a8d; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922
W20260504 19:59:40.340615 12264 leader_election.cc:343] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.341243 12432 leader_election.cc:290] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.341467 12422 raft_consensus.cc:2749] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:40.342175 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.339406 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.343809 12224 raft_consensus.cc:2468] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 0.
I20260504 19:59:40.343900 12431 raft_consensus.cc:515] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.345086 12340 leader_election.cc:304] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.346446 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06a32550992a4221907f6d6a1bd147dd" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
W20260504 19:59:40.346853 12340 leader_election.cc:343] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.347569 12300 raft_consensus.cc:2393] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in current term 1: Already voted for candidate f9a4e75173b54be181f23568394a3a8d in this term.
I20260504 19:59:40.347810 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06a32550992a4221907f6d6a1bd147dd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
W20260504 19:59:40.346669 12189 leader_election.cc:343] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.348541 12224 raft_consensus.cc:3060] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.350096 12191 leader_election.cc:304] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094; no voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:40.352144 12432 raft_consensus.cc:2804] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.355010 12432 raft_consensus.cc:493] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.355659 12432 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.355926 12418 raft_consensus.cc:493] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.356452 12418 raft_consensus.cc:515] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.354359 12413 tablet_bootstrap.cc:654] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.358160 12441 raft_consensus.cc:493] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.358467 12418 leader_election.cc:290] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.358641 12441 raft_consensus.cc:515] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.359206 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06a32550992a4221907f6d6a1bd147dd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:40.360342 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.361001 12431 leader_election.cc:290] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
W20260504 19:59:40.361706 12340 leader_election.cc:343] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.361943 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
W20260504 19:59:40.360443 12264 leader_election.cc:343] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647): Illegal state: must be running to vote when last-logged opid is not known
W20260504 19:59:40.363193 12171 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 19:59:40.363615 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.364702 12441 leader_election.cc:290] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
W20260504 19:59:40.366075 12340 leader_election.cc:343] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.366758 12224 raft_consensus.cc:2468] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 1.
I20260504 19:59:40.366672 12340 leader_election.cc:304] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:40.368075 12266 leader_election.cc:304] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters: bc677a5b14684d43b945885131fe0922
I20260504 19:59:40.368233 12420 raft_consensus.cc:2749] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:40.369362 12431 raft_consensus.cc:2804] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:40.370066 12418 raft_consensus.cc:2749] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:40.371387 12431 raft_consensus.cc:697] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 1 LEADER]: Becoming Leader. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
W20260504 19:59:40.372337 12191 leader_election.cc:343] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.372296 12431 consensus_queue.cc:237] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.372611 12121 catalog_manager.cc:5671] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:40.374171 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
W20260504 19:59:40.375890 12189 leader_election.cc:343] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.376343 12189 leader_election.cc:304] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094; no voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:40.377481 12420 raft_consensus.cc:2749] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:40.380157 12415 tablet_bootstrap.cc:492] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.380784 12415 ts_tablet_manager.cc:1403] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.116s user 0.007s sys 0.017s
I20260504 19:59:40.382273 12414 tablet_bootstrap.cc:492] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.383014 12414 ts_tablet_manager.cc:1403] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.083s user 0.030s sys 0.042s
I20260504 19:59:40.384281 12432 raft_consensus.cc:515] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.384400 12415 raft_consensus.cc:359] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.385388 12415 raft_consensus.cc:385] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.385782 12415 raft_consensus.cc:740] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.386528 12414 raft_consensus.cc:359] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.387370 12414 raft_consensus.cc:385] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.386893 12415 consensus_queue.cc:260] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.387738 12414 raft_consensus.cc:740] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.388721 12414 consensus_queue.cc:260] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.389171 12434 raft_consensus.cc:493] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.389707 12434 raft_consensus.cc:515] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.390275 12415 ts_tablet_manager.cc:1434] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.009s user 0.006s sys 0.005s
I20260504 19:59:40.391637 12415 tablet_bootstrap.cc:492] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.391932 12434 leader_election.cc:290] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.393543 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.394235 12414 ts_tablet_manager.cc:1434] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20260504 19:59:40.394219 12224 raft_consensus.cc:2468] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 0.
I20260504 19:59:40.395382 12414 tablet_bootstrap.cc:492] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.395345 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.397172 12340 leader_election.cc:304] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.397989 12434 raft_consensus.cc:2804] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260504 19:59:40.398135 12340 leader_election.cc:343] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.398378 12434 raft_consensus.cc:493] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.398728 12434 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.407471 12415 tablet_bootstrap.cc:654] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.410432 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:40.410280 12434 raft_consensus.cc:515] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.411203 12224 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.413419 12414 tablet_bootstrap.cc:654] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.416824 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:40.414278 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:40.418362 12225 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.418901 12224 raft_consensus.cc:2468] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 1.
I20260504 19:59:40.420205 12340 leader_election.cc:304] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.421217 12418 raft_consensus.cc:2804] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 19:59:40.424846 12340 leader_election.cc:343] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.432179 12225 raft_consensus.cc:2468] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 1.
I20260504 19:59:40.433233 12434 leader_election.cc:290] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.433933 12340 leader_election.cc:304] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters: f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:40.438253 12432 leader_election.cc:290] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.439124 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:40.436883 12434 raft_consensus.cc:2804] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 19:59:40.440770 12340 leader_election.cc:343] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.440864 12434 raft_consensus.cc:697] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:40.441701 12434 consensus_queue.cc:237] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.446894 12418 raft_consensus.cc:697] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:40.450243 12418 consensus_queue.cc:237] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.453826 12123 catalog_manager.cc:5671] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d reported cstate change: term changed from 0 to 1, leader changed from <none> to f9a4e75173b54be181f23568394a3a8d (127.11.68.2). New cstate: current_term: 1 leader_uuid: "f9a4e75173b54be181f23568394a3a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:40.455118 12414 tablet_bootstrap.cc:492] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.455325 12415 tablet_bootstrap.cc:492] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.455938 12415 ts_tablet_manager.cc:1403] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.065s user 0.024s sys 0.004s
I20260504 19:59:40.456117 12414 ts_tablet_manager.cc:1403] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.017s
I20260504 19:59:40.459156 12415 raft_consensus.cc:359] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.460515 12415 raft_consensus.cc:385] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.460259 12414 raft_consensus.cc:359] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.461544 12414 raft_consensus.cc:385] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.461385 12415 raft_consensus.cc:740] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.462366 12414 raft_consensus.cc:740] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.463754 12415 consensus_queue.cc:260] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.463600 12414 consensus_queue.cc:260] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.467857 12415 ts_tablet_manager.cc:1434] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260504 19:59:40.468364 12414 ts_tablet_manager.cc:1434] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.012s user 0.006s sys 0.003s
I20260504 19:59:40.470815 12415 tablet_bootstrap.cc:492] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.471005 12414 tablet_bootstrap.cc:492] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.479147 12415 tablet_bootstrap.cc:654] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.480619 12414 tablet_bootstrap.cc:654] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.502317 12413 tablet_bootstrap.cc:492] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.503024 12413 ts_tablet_manager.cc:1403] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.232s user 0.018s sys 0.011s
I20260504 19:59:40.510641 12415 tablet_bootstrap.cc:492] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.512051 12414 tablet_bootstrap.cc:492] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.512498 12414 ts_tablet_manager.cc:1403] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.013s
I20260504 19:59:40.514091 12415 ts_tablet_manager.cc:1403] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.008s
I20260504 19:59:40.514221 12413 raft_consensus.cc:359] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.514997 12413 raft_consensus.cc:385] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.515316 12413 raft_consensus.cc:740] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.515161 12414 raft_consensus.cc:359] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.515923 12414 raft_consensus.cc:385] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.516238 12414 raft_consensus.cc:740] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.516127 12413 consensus_queue.cc:260] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.517174 12414 consensus_queue.cc:260] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.517764 12415 raft_consensus.cc:359] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.518750 12415 raft_consensus.cc:385] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.517915 12123 catalog_manager.cc:5671] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:40.519217 12415 raft_consensus.cc:740] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.519954 12415 consensus_queue.cc:260] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.519855 12123 catalog_manager.cc:5671] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:40.521637 12414 ts_tablet_manager.cc:1434] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260504 19:59:40.522163 12415 ts_tablet_manager.cc:1434] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260504 19:59:40.518793 12413 ts_tablet_manager.cc:1434] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260504 19:59:40.522663 12414 tablet_bootstrap.cc:492] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.523272 12413 tablet_bootstrap.cc:492] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.526389 12415 tablet_bootstrap.cc:492] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.530668 12414 tablet_bootstrap.cc:654] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.533610 12415 tablet_bootstrap.cc:654] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.542058 12414 tablet_bootstrap.cc:492] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.542577 12414 ts_tablet_manager.cc:1403] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20260504 19:59:40.544894 12413 tablet_bootstrap.cc:654] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.545408 12414 raft_consensus.cc:359] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.546173 12414 raft_consensus.cc:385] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.546456 12414 raft_consensus.cc:740] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.547139 12414 consensus_queue.cc:260] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.549476 12414 ts_tablet_manager.cc:1434] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260504 19:59:40.550418 12414 tablet_bootstrap.cc:492] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.551193 12415 tablet_bootstrap.cc:492] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.551304 12413 tablet_bootstrap.cc:492] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.551699 12415 ts_tablet_manager.cc:1403] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20260504 19:59:40.552018 12413 ts_tablet_manager.cc:1403] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.001s
I20260504 19:59:40.554873 12415 raft_consensus.cc:359] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.555629 12415 raft_consensus.cc:385] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.555941 12415 raft_consensus.cc:740] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.556274 12414 tablet_bootstrap.cc:654] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.556675 12415 consensus_queue.cc:260] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.559211 12415 ts_tablet_manager.cc:1434] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260504 19:59:40.560570 12415 tablet_bootstrap.cc:492] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.555150 12413 raft_consensus.cc:359] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.562121 12413 raft_consensus.cc:385] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.562563 12413 raft_consensus.cc:740] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.563599 12413 consensus_queue.cc:260] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.566804 12413 ts_tablet_manager.cc:1434] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260504 19:59:40.568199 12413 tablet_bootstrap.cc:492] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.577991 12413 tablet_bootstrap.cc:654] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.579499 12414 tablet_bootstrap.cc:492] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.580027 12414 ts_tablet_manager.cc:1403] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.011s
I20260504 19:59:40.581734 12415 tablet_bootstrap.cc:654] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.582614 12414 raft_consensus.cc:359] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.583280 12413 tablet_bootstrap.cc:492] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.583309 12414 raft_consensus.cc:385] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.583729 12414 raft_consensus.cc:740] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.583863 12413 ts_tablet_manager.cc:1403] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.001s
I20260504 19:59:40.584535 12414 consensus_queue.cc:260] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.586687 12414 ts_tablet_manager.cc:1434] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260504 19:59:40.586931 12415 tablet_bootstrap.cc:492] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.586791 12413 raft_consensus.cc:359] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.587419 12415 ts_tablet_manager.cc:1403] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.001s
I20260504 19:59:40.587493 12413 raft_consensus.cc:385] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.587767 12414 tablet_bootstrap.cc:492] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Bootstrap starting.
I20260504 19:59:40.587877 12413 raft_consensus.cc:740] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.588675 12413 consensus_queue.cc:260] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.590369 12415 raft_consensus.cc:359] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.591060 12415 raft_consensus.cc:385] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.591352 12415 raft_consensus.cc:740] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.592021 12415 consensus_queue.cc:260] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.593663 12414 tablet_bootstrap.cc:654] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.594234 12415 ts_tablet_manager.cc:1434] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260504 19:59:40.595229 12415 tablet_bootstrap.cc:492] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.599287 12413 ts_tablet_manager.cc:1434] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260504 19:59:40.601727 12415 tablet_bootstrap.cc:654] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.603932 12413 tablet_bootstrap.cc:492] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.604700 12414 tablet_bootstrap.cc:492] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: No bootstrap required, opened a new log
I20260504 19:59:40.605185 12414 ts_tablet_manager.cc:1403] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260504 19:59:40.607820 12414 raft_consensus.cc:359] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.608503 12414 raft_consensus.cc:385] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.608807 12414 raft_consensus.cc:740] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.609620 12414 consensus_queue.cc:260] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.610589 12413 tablet_bootstrap.cc:654] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.615413 12414 ts_tablet_manager.cc:1434] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20260504 19:59:40.619453 12415 tablet_bootstrap.cc:492] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.619946 12415 ts_tablet_manager.cc:1403] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.009s
I20260504 19:59:40.622737 12413 tablet_bootstrap.cc:492] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.622705 12415 raft_consensus.cc:359] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.623317 12413 ts_tablet_manager.cc:1403] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20260504 19:59:40.623454 12415 raft_consensus.cc:385] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.623814 12415 raft_consensus.cc:740] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.624488 12415 consensus_queue.cc:260] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.626014 12413 raft_consensus.cc:359] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.626343 12415 ts_tablet_manager.cc:1434] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260504 19:59:40.626528 12413 raft_consensus.cc:385] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.626909 12413 raft_consensus.cc:740] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.627285 12415 tablet_bootstrap.cc:492] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.627548 12413 consensus_queue.cc:260] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.629442 12413 ts_tablet_manager.cc:1434] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260504 19:59:40.630393 12413 tablet_bootstrap.cc:492] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.633699 12415 tablet_bootstrap.cc:654] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.633737 12420 raft_consensus.cc:493] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.634367 12420 raft_consensus.cc:515] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.635879 12413 tablet_bootstrap.cc:654] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.636343 12420 leader_election.cc:290] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.638248 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:40.639005 12372 raft_consensus.cc:2468] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 0.
I20260504 19:59:40.640949 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.641767 12299 raft_consensus.cc:2468] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 0.
I20260504 19:59:40.643973 12191 leader_election.cc:304] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:40.644980 12420 raft_consensus.cc:2804] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.645363 12420 raft_consensus.cc:493] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.645695 12420 raft_consensus.cc:3060] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.647972 12415 tablet_bootstrap.cc:492] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
I20260504 19:59:40.647984 12413 tablet_bootstrap.cc:492] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.648530 12413 ts_tablet_manager.cc:1403] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260504 19:59:40.648530 12415 ts_tablet_manager.cc:1403] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20260504 19:59:40.651480 12415 raft_consensus.cc:359] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.651480 12413 raft_consensus.cc:359] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.652280 12415 raft_consensus.cc:385] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.652307 12413 raft_consensus.cc:385] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.652732 12415 raft_consensus.cc:740] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.652761 12413 raft_consensus.cc:740] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.653488 12415 consensus_queue.cc:260] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.653574 12413 consensus_queue.cc:260] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.653625 12420 raft_consensus.cc:515] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.655857 12413 ts_tablet_manager.cc:1434] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260504 19:59:40.655946 12415 ts_tablet_manager.cc:1434] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260504 19:59:40.655849 12420 leader_election.cc:290] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 election: Requested vote from peers bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.656885 12413 tablet_bootstrap.cc:492] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922: Bootstrap starting.
I20260504 19:59:40.656884 12415 tablet_bootstrap.cc:492] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d: Bootstrap starting.
I20260504 19:59:40.658264 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:40.658454 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:40.658896 12299 raft_consensus.cc:3060] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.658955 12372 raft_consensus.cc:3060] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.663203 12415 tablet_bootstrap.cc:654] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.663316 12413 tablet_bootstrap.cc:654] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922: Neither blocks nor log segments found. Creating new log.
I20260504 19:59:40.672905 12372 raft_consensus.cc:2468] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 1.
I20260504 19:59:40.675509 12299 raft_consensus.cc:2468] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 1.
I20260504 19:59:40.676976 12189 leader_election.cc:304] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.678931 12420 raft_consensus.cc:2804] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:40.679450 12420 raft_consensus.cc:697] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:40.680276 12420 consensus_queue.cc:237] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:40.681591 12413 tablet_bootstrap.cc:492] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922: No bootstrap required, opened a new log
I20260504 19:59:40.682250 12413 ts_tablet_manager.cc:1403] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.011s
I20260504 19:59:40.682857 12434 raft_consensus.cc:493] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.683676 12434 raft_consensus.cc:515] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.686308 12413 raft_consensus.cc:359] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.687036 12413 raft_consensus.cc:385] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.687052 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.687565 12413 raft_consensus.cc:740] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.688638 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.688498 12413 consensus_queue.cc:260] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.689025 12415 tablet_bootstrap.cc:492] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d: No bootstrap required, opened a new log
W20260504 19:59:40.690095 12340 leader_election.cc:343] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Illegal state: must be running to vote when last-logged opid is not known
I20260504 19:59:40.687022 12434 leader_election.cc:290] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.690639 12415 ts_tablet_manager.cc:1403] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.019s
I20260504 19:59:40.691834 12225 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 0.
I20260504 19:59:40.693706 12413 ts_tablet_manager.cc:1434] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260504 19:59:40.694260 12415 raft_consensus.cc:359] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.694873 12415 raft_consensus.cc:385] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 19:59:40.695122 12415 raft_consensus.cc:740] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Initialized, Role: FOLLOWER
I20260504 19:59:40.695050 12340 leader_election.cc:304] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters: f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:40.695926 12415 consensus_queue.cc:260] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.696501 12434 raft_consensus.cc:2804] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.696908 12434 raft_consensus.cc:493] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.697429 12434 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.698495 12415 ts_tablet_manager.cc:1434] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260504 19:59:40.696906 12123 catalog_manager.cc:5671] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:40.709424 12434 raft_consensus.cc:515] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.711138 12434 leader_election.cc:290] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.712060 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:40.712596 12225 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.714164 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:40.714658 12299 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.717202 12225 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 1.
I20260504 19:59:40.718226 12340 leader_election.cc:304] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.719012 12434 raft_consensus.cc:2804] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:40.719408 12434 raft_consensus.cc:697] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:40.719668 12299 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 1.
I20260504 19:59:40.720168 12434 consensus_queue.cc:237] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.727118 12121 catalog_manager.cc:5671] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:40.735512 12431 raft_consensus.cc:493] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.735914 12431 raft_consensus.cc:515] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.737480 12431 leader_election.cc:290] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.737744 12420 raft_consensus.cc:493] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.738267 12420 raft_consensus.cc:515] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.738452 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.738509 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:40.739101 12225 raft_consensus.cc:2468] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 0.
I20260504 19:59:40.739203 12372 raft_consensus.cc:2468] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 0.
I20260504 19:59:40.740170 12420 leader_election.cc:290] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.740321 12264 leader_election.cc:304] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:40.740978 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.741314 12431 raft_consensus.cc:2804] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.741403 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:40.741601 12299 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 0.
I20260504 19:59:40.741742 12431 raft_consensus.cc:493] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.742115 12431 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.742061 12372 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 0.
I20260504 19:59:40.742713 12191 leader_election.cc:304] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:40.743557 12420 raft_consensus.cc:2804] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.743916 12420 raft_consensus.cc:493] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.744256 12420 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.748922 12434 raft_consensus.cc:493] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.749380 12434 raft_consensus.cc:515] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.749732 12431 raft_consensus.cc:515] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.751121 12434 leader_election.cc:290] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.751284 12420 raft_consensus.cc:515] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.751829 12431 leader_election.cc:290] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.752988 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:40.753265 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06a32550992a4221907f6d6a1bd147dd" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.753587 12224 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.754251 12225 raft_consensus.cc:2393] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc677a5b14684d43b945885131fe0922 in current term 1: Already voted for candidate f9a4e75173b54be181f23568394a3a8d in this term.
I20260504 19:59:40.754616 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:40.755175 12300 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.755308 12298 raft_consensus.cc:3060] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.755851 12420 leader_election.cc:290] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.755331 12431 raft_consensus.cc:493] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.757345 12431 raft_consensus.cc:515] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.759618 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:40.759819 12431 leader_election.cc:290] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.760234 12373 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.761221 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:40.761895 12372 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.762730 12374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:40.763656 12300 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 1.
I20260504 19:59:40.764854 12191 leader_election.cc:304] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:40.752988 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06a32550992a4221907f6d6a1bd147dd" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.765750 12420 raft_consensus.cc:2804] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:40.766201 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.766255 12434 consensus_queue.cc:1048] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:40.767071 12225 raft_consensus.cc:2393] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9a4e75173b54be181f23568394a3a8d in current term 1: Already voted for candidate bc677a5b14684d43b945885131fe0922 in this term.
I20260504 19:59:40.766989 12420 raft_consensus.cc:697] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:40.767926 12420 consensus_queue.cc:237] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.768903 12264 leader_election.cc:304] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a4e75173b54be181f23568394a3a8d; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922
I20260504 19:59:40.769447 12340 leader_election.cc:304] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:40.770207 12418 raft_consensus.cc:493] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.770056 12422 consensus_queue.cc:1048] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:40.770728 12418 raft_consensus.cc:515] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.771524 12422 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.771602 12224 raft_consensus.cc:2468] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 1.
I20260504 19:59:40.772650 12418 leader_election.cc:290] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.773026 12434 raft_consensus.cc:3060] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.773914 12266 leader_election.cc:304] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:40.777328 12431 raft_consensus.cc:2804] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:40.778062 12431 raft_consensus.cc:697] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 1 LEADER]: Becoming Leader. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
I20260504 19:59:40.778920 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.779624 12225 raft_consensus.cc:2468] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 0.
I20260504 19:59:40.778920 12431 consensus_queue.cc:237] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.780733 12340 leader_election.cc:304] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.781526 12418 raft_consensus.cc:2804] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.781908 12418 raft_consensus.cc:493] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.782248 12418 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.780074 12298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.785722 12372 raft_consensus.cc:2468] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 1.
I20260504 19:59:40.787333 12298 raft_consensus.cc:2468] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 0.
I20260504 19:59:40.791097 12422 raft_consensus.cc:2749] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:40.801673 12418 raft_consensus.cc:515] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.805445 12225 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:40.806135 12225 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.808032 12448 consensus_queue.cc:1048] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:40.789263 12373 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 1.
I20260504 19:59:40.814123 12434 raft_consensus.cc:2749] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 19:59:40.816478 12225 raft_consensus.cc:2468] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 1.
I20260504 19:59:40.818006 12340 leader_election.cc:304] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.820344 12298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:40.820976 12298 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.821907 12418 leader_election.cc:290] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 1 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:40.822613 12418 raft_consensus.cc:2804] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:40.823199 12418 raft_consensus.cc:697] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:40.824283 12418 consensus_queue.cc:237] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.821509 12121 catalog_manager.cc:5671] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:40.851104 12298 raft_consensus.cc:2468] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 1.
I20260504 19:59:40.853942 12123 catalog_manager.cc:5671] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d reported cstate change: term changed from 0 to 1, leader changed from <none> to f9a4e75173b54be181f23568394a3a8d (127.11.68.2). New cstate: current_term: 1 leader_uuid: "f9a4e75173b54be181f23568394a3a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:40.876116 12469 consensus_queue.cc:1048] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:40.886731 12123 catalog_manager.cc:5671] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:40.890362 12299 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.901005 12431 raft_consensus.cc:493] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.901520 12431 raft_consensus.cc:515] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.902177 12418 consensus_queue.cc:1048] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:40.905639 12431 leader_election.cc:290] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.906189 12223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:40.906288 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:40.906904 12223 raft_consensus.cc:2468] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 1.
I20260504 19:59:40.907976 12266 leader_election.cc:304] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:40.909709 12431 raft_consensus.cc:2804] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 19:59:40.910518 12431 raft_consensus.cc:493] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Starting leader election (detected failure of leader bc677a5b14684d43b945885131fe0922)
I20260504 19:59:40.910959 12431 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:40.917718 12434 consensus_queue.cc:1048] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:40.918382 12431 raft_consensus.cc:515] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.919935 12299 raft_consensus.cc:1240] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Rejecting Update request from peer bc677a5b14684d43b945885131fe0922 for earlier term 1. Current term is 2. Ops: []
I20260504 19:59:40.921631 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:40.923355 12431 leader_election.cc:290] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 2 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.924084 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:40.925982 12264 leader_election.cc:304] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a4e75173b54be181f23568394a3a8d; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922
I20260504 19:59:40.926775 12431 raft_consensus.cc:2749] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 19:59:40.929122 12434 consensus_queue.cc:1059] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:40.929795 12448 raft_consensus.cc:3055] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:40.930197 12448 raft_consensus.cc:740] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:40.931026 12448 consensus_queue.cc:260] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.932476 12448 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:40.953317 12420 raft_consensus.cc:493] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 19:59:40.954051 12420 raft_consensus.cc:515] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.955857 12420 leader_election.cc:290] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.956748 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d2807b93ae4423aeeeae7b5334de70" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:40.956892 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d2807b93ae4423aeeeae7b5334de70" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:40.957505 12299 raft_consensus.cc:2468] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 0.
I20260504 19:59:40.957505 12373 raft_consensus.cc:2468] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 0.
I20260504 19:59:40.958825 12191 leader_election.cc:304] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:40.959517 12420 raft_consensus.cc:2804] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 19:59:40.959791 12420 raft_consensus.cc:493] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 19:59:40.960124 12420 raft_consensus.cc:3060] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.962157 12448 consensus_queue.cc:1048] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:40.967070 12420 raft_consensus.cc:515] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.968845 12420 leader_election.cc:290] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:40.970216 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d2807b93ae4423aeeeae7b5334de70" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:40.970417 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d2807b93ae4423aeeeae7b5334de70" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:40.970839 12299 raft_consensus.cc:3060] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.970903 12373 raft_consensus.cc:3060] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 0 FOLLOWER]: Advancing to term 1
I20260504 19:59:40.976922 12432 consensus_queue.cc:1048] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:40.977772 12373 raft_consensus.cc:2468] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 1.
I20260504 19:59:40.979640 12189 leader_election.cc:304] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:40.977526 12299 raft_consensus.cc:2468] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 1.
I20260504 19:59:40.980599 12420 raft_consensus.cc:2804] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 19:59:40.981528 12420 raft_consensus.cc:697] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:40.982643 12420 consensus_queue.cc:237] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:40.993269 12121 catalog_manager.cc:5671] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:41.109310 12373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0bbfa5c44a6d4d61a6175fe1e4a62d53"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:47786
I20260504 19:59:41.109977 12373 raft_consensus.cc:606] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:41.114918 12373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:47786
I20260504 19:59:41.115455 12373 raft_consensus.cc:606] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:41.120216 12373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:47786
I20260504 19:59:41.120724 12373 raft_consensus.cc:606] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:41.121021 12373 raft_consensus.cc:614] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 19:59:41.125409 12373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:47786
I20260504 19:59:41.125944 12373 raft_consensus.cc:606] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:41.134878 12420 consensus_queue.cc:1048] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.140964 12223 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
mode: GRACEFUL
new_leader_uuid: "f9a4e75173b54be181f23568394a3a8d"
from {username='slave'} at 127.0.0.1:59776
I20260504 19:59:41.141613 12223 raft_consensus.cc:606] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Received request to transfer leadership to TS f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:41.149029 12491 consensus_queue.cc:1048] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:41.151162 12373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "734e7619181643f7a33111df773337b0"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
mode: GRACEFUL
new_leader_uuid: "f9a4e75173b54be181f23568394a3a8d"
from {username='slave'} at 127.0.0.1:47786
I20260504 19:59:41.151851 12373 raft_consensus.cc:606] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Received request to transfer leadership to TS f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:41.165210 12223 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27d2807b93ae4423aeeeae7b5334de70"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
mode: GRACEFUL
new_leader_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:59776
I20260504 19:59:41.165855 12223 raft_consensus.cc:606] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Received request to transfer leadership to TS bc677a5b14684d43b945885131fe0922
I20260504 19:59:41.170527 12491 consensus_queue.cc:1048] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 19:59:41.181888 12420 consensus_queue.cc:1048] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.190078 12441 raft_consensus.cc:993] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: : Instructing follower f9a4e75173b54be181f23568394a3a8d to start an election
I20260504 19:59:41.190564 12420 raft_consensus.cc:1081] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Signalling peer f9a4e75173b54be181f23568394a3a8d to start an election
I20260504 19:59:41.190354 12469 consensus_queue.cc:1048] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.192155 12298 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426"
dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
from {username='slave'} at 127.0.0.1:55378
I20260504 19:59:41.192749 12298 raft_consensus.cc:493] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:41.193106 12298 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.200910 12298 raft_consensus.cc:515] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.205457 12298 leader_election.cc:290] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 2 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:41.206319 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:41.206911 12373 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.209280 12223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:41.209933 12223 raft_consensus.cc:3055] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:41.210263 12223 raft_consensus.cc:740] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:41.211018 12223 consensus_queue.cc:260] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.211633 12431 consensus_queue.cc:1048] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.213057 12432 raft_consensus.cc:993] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922: : Instructing follower 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:41.213479 12432 raft_consensus.cc:1081] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Signalling peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:41.213956 12373 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 2.
I20260504 19:59:41.215241 12264 leader_election.cc:304] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:41.215296 12224 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0bbfa5c44a6d4d61a6175fe1e4a62d53"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:59756
I20260504 19:59:41.216001 12224 raft_consensus.cc:493] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:41.216374 12224 raft_consensus.cc:3060] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.219640 12431 raft_consensus.cc:2804] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:41.220156 12431 raft_consensus.cc:697] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Becoming Leader. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
I20260504 19:59:41.220986 12431 consensus_queue.cc:237] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.222440 12223 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.229923 12224 raft_consensus.cc:515] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.232046 12224 leader_election.cc:290] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:41.232743 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bbfa5c44a6d4d61a6175fe1e4a62d53" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:41.233314 12299 raft_consensus.cc:3060] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.233304 12434 consensus_queue.cc:1048] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.234246 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bbfa5c44a6d4d61a6175fe1e4a62d53" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:41.233870 12120 catalog_manager.cc:5671] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d reported cstate change: term changed from 1 to 2, leader changed from 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1) to f9a4e75173b54be181f23568394a3a8d (127.11.68.2). New cstate: current_term: 2 leader_uuid: "f9a4e75173b54be181f23568394a3a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:41.234825 12373 raft_consensus.cc:3055] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:41.235172 12373 raft_consensus.cc:740] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:41.235927 12373 consensus_queue.cc:260] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.237236 12373 raft_consensus.cc:3060] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.242169 12223 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 2.
I20260504 19:59:41.249393 12373 raft_consensus.cc:2468] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:41.250797 12189 leader_election.cc:304] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:41.250923 12299 raft_consensus.cc:2468] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:41.251657 12420 raft_consensus.cc:2804] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:41.252141 12420 raft_consensus.cc:697] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:41.252959 12420 consensus_queue.cc:237] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.256551 12432 consensus_queue.cc:1048] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.267021 12121 catalog_manager.cc:5671] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 1 to 2, leader changed from bc677a5b14684d43b945885131fe0922 (127.11.68.3) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:41.268396 12434 raft_consensus.cc:993] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922: : Instructing follower 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:41.268882 12432 raft_consensus.cc:1081] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Signalling peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:41.270430 12223 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:59756
I20260504 19:59:41.271009 12223 raft_consensus.cc:493] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:41.271384 12223 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.277963 12223 raft_consensus.cc:515] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.280303 12223 leader_election.cc:290] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:41.282086 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:41.282687 12299 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.282802 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:41.283438 12373 raft_consensus.cc:3055] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:41.283921 12373 raft_consensus.cc:740] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:41.284860 12373 consensus_queue.cc:260] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.286079 12373 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.289028 12299 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:41.290038 12191 leader_election.cc:304] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:41.290695 12420 raft_consensus.cc:2804] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:41.291076 12420 raft_consensus.cc:697] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:41.292069 12373 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:41.291831 12420 consensus_queue.cc:237] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.298166 12122 catalog_manager.cc:5671] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 1 to 2, leader changed from bc677a5b14684d43b945885131fe0922 (127.11.68.3) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:41.378921 12434 raft_consensus.cc:993] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922: : Instructing follower 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:41.379384 12432 raft_consensus.cc:1081] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Signalling peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:41.379577 12420 consensus_queue.cc:1048] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.380879 12223 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:59756
I20260504 19:59:41.381456 12223 raft_consensus.cc:493] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:41.381767 12223 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.388458 12223 raft_consensus.cc:515] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.391261 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:41.391873 12300 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.392288 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:41.392891 12372 raft_consensus.cc:3055] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:41.393216 12372 raft_consensus.cc:740] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:41.394011 12372 consensus_queue.cc:260] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.395099 12372 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.396441 12491 consensus_queue.cc:1048] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.399241 12300 raft_consensus.cc:2468] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:41.400441 12191 leader_election.cc:304] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:41.401351 12441 raft_consensus.cc:2804] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:41.401824 12441 raft_consensus.cc:697] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:41.402653 12441 consensus_queue.cc:237] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.405193 12420 raft_consensus.cc:993] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: : Instructing follower bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:41.405632 12491 raft_consensus.cc:1081] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Signalling peer bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:41.406646 12223 leader_election.cc:290] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:41.407053 12373 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "27d2807b93ae4423aeeeae7b5334de70"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:47772
I20260504 19:59:41.407593 12373 raft_consensus.cc:493] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:41.408599 12373 raft_consensus.cc:3060] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.410866 12372 raft_consensus.cc:2468] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:41.414887 12373 raft_consensus.cc:515] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.416350 12432 consensus_queue.cc:1048] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.415952 12122 catalog_manager.cc:5671] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 1 to 2, leader changed from bc677a5b14684d43b945885131fe0922 (127.11.68.3) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:41.418541 12373 leader_election.cc:290] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 2 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:41.419945 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d2807b93ae4423aeeeae7b5334de70" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:41.420495 12299 raft_consensus.cc:3060] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.421495 12223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27d2807b93ae4423aeeeae7b5334de70" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:41.422154 12223 raft_consensus.cc:3055] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:41.422488 12223 raft_consensus.cc:740] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:41.423249 12223 consensus_queue.cc:260] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.424489 12223 raft_consensus.cc:3060] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.427683 12299 raft_consensus.cc:2468] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 2.
I20260504 19:59:41.429180 12340 leader_election.cc:304] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:41.431478 12432 raft_consensus.cc:2804] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:41.431869 12223 raft_consensus.cc:2468] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 2.
I20260504 19:59:41.431968 12432 raft_consensus.cc:697] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 2 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:41.432825 12418 raft_consensus.cc:993] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922: : Instructing follower f9a4e75173b54be181f23568394a3a8d to start an election
I20260504 19:59:41.432950 12432 consensus_queue.cc:237] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.434332 12448 raft_consensus.cc:1081] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Signalling peer f9a4e75173b54be181f23568394a3a8d to start an election
I20260504 19:59:41.439658 12300 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "734e7619181643f7a33111df773337b0"
dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
from {username='slave'} at 127.0.0.1:55376
I20260504 19:59:41.440090 12418 consensus_queue.cc:1048] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:41.441279 12300 raft_consensus.cc:493] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:41.441730 12300 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.442371 12122 catalog_manager.cc:5671] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 1 to 2, leader changed from 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1) to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:41.450225 12300 raft_consensus.cc:515] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.456166 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:41.456881 12224 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.459558 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:41.460141 12373 raft_consensus.cc:3055] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:41.460461 12373 raft_consensus.cc:740] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:41.462090 12300 leader_election.cc:290] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 2 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:41.461386 12373 consensus_queue.cc:260] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.464138 12224 raft_consensus.cc:2468] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 2.
I20260504 19:59:41.464211 12373 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:41.465312 12266 leader_election.cc:304] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:41.466056 12431 raft_consensus.cc:2804] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:41.466425 12431 raft_consensus.cc:697] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Becoming Leader. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
I20260504 19:59:41.467202 12431 consensus_queue.cc:237] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:41.471792 12373 raft_consensus.cc:2468] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 2.
I20260504 19:59:41.474504 12121 catalog_manager.cc:5671] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d reported cstate change: term changed from 1 to 2, leader changed from bc677a5b14684d43b945885131fe0922 (127.11.68.3) to f9a4e75173b54be181f23568394a3a8d (127.11.68.2). New cstate: current_term: 2 leader_uuid: "f9a4e75173b54be181f23568394a3a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:41.639132 12373 raft_consensus.cc:1275] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.640494 12441 consensus_queue.cc:1048] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:41.650147 12224 raft_consensus.cc:1275] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Refusing update from remote peer f9a4e75173b54be181f23568394a3a8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.650554 12300 raft_consensus.cc:1275] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.651876 12431 consensus_queue.cc:1048] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:41.652109 12441 consensus_queue.cc:1048] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:41.661944 12373 raft_consensus.cc:1275] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Refusing update from remote peer f9a4e75173b54be181f23568394a3a8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.663947 12431 consensus_queue.cc:1048] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 19:59:41.703495 12300 raft_consensus.cc:1275] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.704700 12441 consensus_queue.cc:1048] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:41.712124 12373 raft_consensus.cc:1275] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.713640 12491 consensus_queue.cc:1048] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:41.884809 12373 raft_consensus.cc:1275] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.886193 12441 consensus_queue.cc:1048] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:41.894523 12373 raft_consensus.cc:1275] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Refusing update from remote peer f9a4e75173b54be181f23568394a3a8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.895228 12300 raft_consensus.cc:1275] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.896202 12422 consensus_queue.cc:1048] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:41.898005 12491 consensus_queue.cc:1048] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:41.907414 12223 raft_consensus.cc:1275] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Refusing update from remote peer f9a4e75173b54be181f23568394a3a8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:41.908830 12422 consensus_queue.cc:1048] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:42.032589 12300 raft_consensus.cc:1275] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:42.033823 12418 consensus_queue.cc:1048] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:42.042410 12223 raft_consensus.cc:1275] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:42.043984 12505 consensus_queue.cc:1048] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:42.421415 12420 raft_consensus.cc:493] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc677a5b14684d43b945885131fe0922)
I20260504 19:59:42.421999 12420 raft_consensus.cc:515] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:42.423692 12420 leader_election.cc:290] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:42.424690 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d" is_pre_election: true
I20260504 19:59:42.425489 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:42.426460 12300 raft_consensus.cc:2393] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in current term 2: Already voted for candidate f9a4e75173b54be181f23568394a3a8d in this term.
I20260504 19:59:42.426405 12372 raft_consensus.cc:2468] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:42.427793 12189 leader_election.cc:304] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters: f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:42.428498 12420 raft_consensus.cc:2804] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 19:59:42.428846 12420 raft_consensus.cc:493] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bc677a5b14684d43b945885131fe0922)
I20260504 19:59:42.429145 12420 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:42.434607 12420 raft_consensus.cc:515] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:42.436829 12420 leader_election.cc:290] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:42.437206 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:42.438093 12300 raft_consensus.cc:2393] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in current term 2: Already voted for candidate f9a4e75173b54be181f23568394a3a8d in this term.
I20260504 19:59:42.438679 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:42.446149 12372 raft_consensus.cc:2468] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:42.447139 12189 leader_election.cc:304] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters: f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:42.447925 12420 raft_consensus.cc:2804] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:42.448449 12420 raft_consensus.cc:697] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:42.449178 12420 consensus_queue.cc:237] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:42.456475 12120 catalog_manager.cc:5671] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 1 to 2, leader changed from bc677a5b14684d43b945885131fe0922 (127.11.68.3) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:42.864595 12300 raft_consensus.cc:1275] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:42.866034 12420 consensus_queue.cc:1048] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 19:59:42.869359 12373 raft_consensus.cc:1275] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:42.871024 12420 consensus_queue.cc:1048] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:42.871995 12469 raft_consensus.cc:493] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f2b1ad3a61b46e98ebd6aa6ac8b4094)
I20260504 19:59:42.872463 12469 raft_consensus.cc:515] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:42.876660 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bc677a5b14684d43b945885131fe0922" is_pre_election: true
I20260504 19:59:42.886215 12469 leader_election.cc:290] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:42.887989 12223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" is_pre_election: true
I20260504 19:59:42.889662 12266 leader_election.cc:304] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a4e75173b54be181f23568394a3a8d; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922
I20260504 19:59:42.890794 12469 raft_consensus.cc:2749] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 19:59:44.200579 11536 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20260504 19:59:44.201333 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:44.218400 12223 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
mode: GRACEFUL
new_leader_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:50344
I20260504 19:59:44.218976 12223 raft_consensus.cc:606] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Received request to transfer leadership to TS bc677a5b14684d43b945885131fe0922
I20260504 19:59:44.221565 12223 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
mode: GRACEFUL
new_leader_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:50344
I20260504 19:59:44.222241 12223 raft_consensus.cc:606] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Received request to transfer leadership to TS bc677a5b14684d43b945885131fe0922
I20260504 19:59:44.223330 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260504 19:59:44.223904 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:44.394078 12519 raft_consensus.cc:993] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: : Instructing follower bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:44.394621 12441 raft_consensus.cc:1081] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Signalling peer bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:44.396096 12373 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:47772
I20260504 19:59:44.396564 12373 raft_consensus.cc:493] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:44.396865 12373 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:44.402256 12373 raft_consensus.cc:515] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:44.403795 12373 leader_election.cc:290] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:44.404798 12223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:44.405084 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:44.405365 12223 raft_consensus.cc:3055] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:44.405721 12223 raft_consensus.cc:740] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:44.405746 12300 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:44.406666 12223 consensus_queue.cc:260] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:44.407627 12223 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:44.412582 12223 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 3.
I20260504 19:59:44.413444 12300 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 3.
I20260504 19:59:44.413715 12340 leader_election.cc:304] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:44.414615 12505 raft_consensus.cc:2804] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:44.415710 12505 raft_consensus.cc:697] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:44.416620 12505 consensus_queue.cc:237] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:44.424945 12121 catalog_manager.cc:5671] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 2 to 3, leader changed from 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1) to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:44.650588 12519 raft_consensus.cc:993] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: : Instructing follower bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:44.651027 12441 raft_consensus.cc:1081] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Signalling peer bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:44.652456 12373 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:47772
I20260504 19:59:44.652946 12373 raft_consensus.cc:493] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:44.653234 12373 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:44.658154 12373 raft_consensus.cc:515] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:44.659549 12373 leader_election.cc:290] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:44.660549 12223 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:44.660748 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:44.661074 12223 raft_consensus.cc:3055] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:44.661298 12300 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:44.661417 12223 raft_consensus.cc:740] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:44.662220 12223 consensus_queue.cc:260] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:44.663200 12223 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:44.666921 12300 raft_consensus.cc:2468] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 3.
I20260504 19:59:44.667953 12340 leader_election.cc:304] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:44.668164 12223 raft_consensus.cc:2468] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 3.
I20260504 19:59:44.668803 12505 raft_consensus.cc:2804] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:44.669327 12505 raft_consensus.cc:697] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:44.670112 12505 consensus_queue.cc:237] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:44.677062 12121 catalog_manager.cc:5671] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 2 to 3, leader changed from 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1) to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:44.952518 12223 raft_consensus.cc:1275] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:44.954131 12505 consensus_queue.cc:1048] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:44.966483 12300 raft_consensus.cc:1275] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:44.969432 12520 consensus_queue.cc:1048] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:45.225231 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:45.232044 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:45.232491 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:45.250730 12223 raft_consensus.cc:1275] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:45.252275 12505 consensus_queue.cc:1048] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:45.253989 12300 raft_consensus.cc:1275] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:45.255584 12505 consensus_queue.cc:1048] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:46.233726 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:46.239452 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:46.239872 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:47.240890 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:47.246124 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:47.246503 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:48.247356 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:48.253558 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:48.254046 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:49.255172 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:49.261447 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:49.262152 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:50.263136 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:50.270344 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:50.270723 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:51.271687 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:51.276991 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:51.277628 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:52.278640 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:52.283864 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:52.284281 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:53.285115 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:53.290313 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:53.290690 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:54.323875 11536 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20260504 19:59:54.333576 12373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:47786
I20260504 19:59:54.334280 12373 raft_consensus.cc:606] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:54.341892 12373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:47786
I20260504 19:59:54.342396 12373 raft_consensus.cc:606] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:54.358382 12300 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426"
dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:57524
I20260504 19:59:54.358955 12300 raft_consensus.cc:606] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:54.364048 12300 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "734e7619181643f7a33111df773337b0"
dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:57524
I20260504 19:59:54.364559 12300 raft_consensus.cc:606] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:54.369196 12300 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952"
dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
mode: GRACEFUL
new_leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:57524
I20260504 19:59:54.369696 12300 raft_consensus.cc:606] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 1 LEADER]: Received request to transfer leadership to TS 7f2b1ad3a61b46e98ebd6aa6ac8b4094
I20260504 19:59:54.416282 12542 raft_consensus.cc:993] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d: : Instructing follower 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.416999 12539 raft_consensus.cc:1081] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Signalling peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.418679 12223 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "734e7619181643f7a33111df773337b0"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:59772
I20260504 19:59:54.419168 12223 raft_consensus.cc:493] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:54.419481 12223 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:54.424749 12223 raft_consensus.cc:515] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.426501 12223 leader_election.cc:290] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 3 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:54.427420 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:54.427680 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "734e7619181643f7a33111df773337b0" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:54.428156 12300 raft_consensus.cc:3055] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:54.428256 12373 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:54.428452 12300 raft_consensus.cc:740] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
I20260504 19:59:54.429198 12300 consensus_queue.cc:260] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.430271 12300 raft_consensus.cc:3060] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:54.433296 12373 raft_consensus.cc:2468] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 3.
I20260504 19:59:54.434646 12189 leader_election.cc:304] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:54.435585 12441 raft_consensus.cc:2804] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:54.435685 12300 raft_consensus.cc:2468] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 3.
I20260504 19:59:54.436640 12441 raft_consensus.cc:697] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:54.437605 12441 consensus_queue.cc:237] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.444792 12121 catalog_manager.cc:5671] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 2 to 3, leader changed from f9a4e75173b54be181f23568394a3a8d (127.11.68.2) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:54.566164 12505 raft_consensus.cc:993] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922: : Instructing follower 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.566675 12549 raft_consensus.cc:1081] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Signalling peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.568061 12223 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:59756
I20260504 19:59:54.568539 12223 raft_consensus.cc:493] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:54.568823 12223 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:54.573163 12223 raft_consensus.cc:515] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.574492 12223 leader_election.cc:290] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 4 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:54.575547 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:54.575555 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2fd6cd6b68b47f999acb5ea52fde71e" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:54.576061 12300 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:54.576143 12373 raft_consensus.cc:3055] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 19:59:54.576467 12373 raft_consensus.cc:740] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:54.577155 12373 consensus_queue.cc:260] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.578181 12373 raft_consensus.cc:3060] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:54.580780 12300 raft_consensus.cc:2468] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 4.
I20260504 19:59:54.581883 12191 leader_election.cc:304] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:54.582598 12441 raft_consensus.cc:2804] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 19:59:54.582945 12441 raft_consensus.cc:697] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:54.583107 12373 raft_consensus.cc:2468] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 4.
I20260504 19:59:54.583614 12441 consensus_queue.cc:237] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.590081 12121 catalog_manager.cc:5671] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 3 to 4, leader changed from bc677a5b14684d43b945885131fe0922 (127.11.68.3) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 4 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:54.597133 12542 raft_consensus.cc:993] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d: : Instructing follower 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.597636 12539 raft_consensus.cc:1081] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Signalling peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.598776 12539 raft_consensus.cc:993] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d: : Instructing follower 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.599236 12542 raft_consensus.cc:1081] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 1 LEADER]: Signalling peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.599332 12223 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:59772
I20260504 19:59:54.599864 12223 raft_consensus.cc:493] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:54.600225 12223 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:54.601128 12224 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:59772
I20260504 19:59:54.601657 12224 raft_consensus.cc:493] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:54.602028 12224 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:54.607185 12223 raft_consensus.cc:515] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.609076 12223 leader_election.cc:290] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 3 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:54.608919 12224 raft_consensus.cc:515] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.609995 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:54.610137 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:54.610596 12300 raft_consensus.cc:3055] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:54.610688 12373 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:54.610967 12300 raft_consensus.cc:740] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
I20260504 19:59:54.611872 12224 leader_election.cc:290] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:54.611907 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:54.612613 12299 raft_consensus.cc:3055] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:54.612579 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:54.613067 12299 raft_consensus.cc:740] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
I20260504 19:59:54.613296 12372 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:54.611819 12300 consensus_queue.cc:260] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.613945 12299 consensus_queue.cc:260] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.614569 12300 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:54.615264 12299 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:54.617661 12373 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 3.
I20260504 19:59:54.618770 12189 leader_election.cc:304] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:54.619664 12441 raft_consensus.cc:2804] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:54.620079 12441 raft_consensus.cc:697] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:54.620766 12372 raft_consensus.cc:2468] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:54.620852 12441 consensus_queue.cc:237] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.621582 12300 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 3.
I20260504 19:59:54.623054 12189 leader_election.cc:304] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, bc677a5b14684d43b945885131fe0922; no voters:
I20260504 19:59:54.623248 12299 raft_consensus.cc:2468] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 2.
I20260504 19:59:54.624112 12543 raft_consensus.cc:2804] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:54.625623 12543 raft_consensus.cc:697] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:54.626580 12543 consensus_queue.cc:237] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.629383 12121 catalog_manager.cc:5671] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 2 to 3, leader changed from f9a4e75173b54be181f23568394a3a8d (127.11.68.2) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:54.640520 12121 catalog_manager.cc:5671] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 1 to 2, leader changed from f9a4e75173b54be181f23568394a3a8d (127.11.68.2) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:54.689401 12505 raft_consensus.cc:993] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922: : Instructing follower 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.689850 12549 raft_consensus.cc:1081] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Signalling peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 to start an election
I20260504 19:59:54.691211 12224 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
from {username='slave'} at 127.0.0.1:59756
I20260504 19:59:54.691795 12224 raft_consensus.cc:493] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:54.692152 12224 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:54.697674 12224 raft_consensus.cc:515] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.699295 12224 leader_election.cc:290] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 4 election: Requested vote from peers f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:54.700312 12299 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:54.700343 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdaee4dfea3548dcb8579e74423edb9c" candidate_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:54.701000 12299 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:54.701089 12372 raft_consensus.cc:3055] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 19:59:54.701470 12372 raft_consensus.cc:740] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:54.702181 12372 consensus_queue.cc:260] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.703166 12372 raft_consensus.cc:3060] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:54.706363 12299 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 4.
I20260504 19:59:54.707379 12191 leader_election.cc:304] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:54.708011 12372 raft_consensus.cc:2468] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f2b1ad3a61b46e98ebd6aa6ac8b4094 in term 4.
I20260504 19:59:54.708096 12543 raft_consensus.cc:2804] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 19:59:54.708736 12543 raft_consensus.cc:697] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 LEADER]: Becoming Leader. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:54.709554 12543 consensus_queue.cc:237] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:54.716468 12121 catalog_manager.cc:5671] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 reported cstate change: term changed from 3 to 4, leader changed from bc677a5b14684d43b945885131fe0922 (127.11.68.3) to 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1). New cstate: current_term: 4 leader_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:54.898411 12299 raft_consensus.cc:1275] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:54.899895 12543 consensus_queue.cc:1048] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:54.909760 12372 raft_consensus.cc:1275] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:54.912322 12543 consensus_queue.cc:1048] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:55.060621 12300 raft_consensus.cc:1275] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:55.061930 12543 consensus_queue.cc:1048] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:55.071346 12372 raft_consensus.cc:1275] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:55.071647 12300 raft_consensus.cc:1275] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:55.072939 12543 consensus_queue.cc:1048] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:55.073567 12441 consensus_queue.cc:1048] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:55.088481 12373 raft_consensus.cc:1275] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:55.089890 12543 consensus_queue.cc:1048] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:55.134773 12373 raft_consensus.cc:1275] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:55.135969 12543 consensus_queue.cc:1048] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:55.144397 12299 raft_consensus.cc:1275] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:55.146059 12441 consensus_queue.cc:1048] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:55.179134 12299 raft_consensus.cc:1275] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:55.180567 12441 consensus_queue.cc:1048] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:55.189625 12373 raft_consensus.cc:1275] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Refusing update from remote peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:55.191020 12552 consensus_queue.cc:1048] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 19:59:57.407002 11536 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20260504 19:59:57.407804 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:57.415419 12224 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
mode: GRACEFUL
new_leader_uuid: "f9a4e75173b54be181f23568394a3a8d"
from {username='slave'} at 127.0.0.1:50344
I20260504 19:59:57.416103 12224 raft_consensus.cc:606] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 LEADER]: Received request to transfer leadership to TS f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:57.418642 12224 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
mode: GRACEFUL
new_leader_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:50344
I20260504 19:59:57.419062 12224 raft_consensus.cc:606] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Received request to transfer leadership to TS bc677a5b14684d43b945885131fe0922
I20260504 19:59:57.421020 12224 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
mode: GRACEFUL
new_leader_uuid: "f9a4e75173b54be181f23568394a3a8d"
from {username='slave'} at 127.0.0.1:50344
I20260504 19:59:57.421468 12224 raft_consensus.cc:606] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Received request to transfer leadership to TS f9a4e75173b54be181f23568394a3a8d
I20260504 19:59:57.423653 12224 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd"
dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
mode: GRACEFUL
new_leader_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:50344
I20260504 19:59:57.424052 12224 raft_consensus.cc:606] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Received request to transfer leadership to TS bc677a5b14684d43b945885131fe0922
I20260504 19:59:57.424980 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260504 19:59:57.425465 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:57.461220 12441 raft_consensus.cc:993] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: : Instructing follower f9a4e75173b54be181f23568394a3a8d to start an election
I20260504 19:59:57.461758 12441 raft_consensus.cc:1081] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 LEADER]: Signalling peer f9a4e75173b54be181f23568394a3a8d to start an election
I20260504 19:59:57.463791 12300 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426"
dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
from {username='slave'} at 127.0.0.1:55378
I20260504 19:59:57.464521 12300 raft_consensus.cc:493] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:57.464910 12300 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:57.470444 12300 raft_consensus.cc:515] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.472529 12300 leader_election.cc:290] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 4 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:57.473480 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:57.473986 12372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd1d29b2cc94b5390cc815b5287f426" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:57.474320 12224 raft_consensus.cc:3055] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 19:59:57.474557 12372 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:57.474785 12224 raft_consensus.cc:740] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:57.475798 12224 consensus_queue.cc:260] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.477205 12224 raft_consensus.cc:3060] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Advancing to term 4
I20260504 19:59:57.478936 12570 raft_consensus.cc:993] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: : Instructing follower bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:57.479491 12441 raft_consensus.cc:1081] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Signalling peer bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:57.481204 12373 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:47772
I20260504 19:59:57.481750 12373 raft_consensus.cc:493] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:57.482128 12373 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:57.482264 12372 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 4.
I20260504 19:59:57.483476 12264 leader_election.cc:304] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:57.484428 12542 raft_consensus.cc:2804] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Leader election won for term 4
I20260504 19:59:57.485532 12224 raft_consensus.cc:2468] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 4.
I20260504 19:59:57.485625 12542 raft_consensus.cc:697] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 4 LEADER]: Becoming Leader. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
I20260504 19:59:57.487497 12542 consensus_queue.cc:237] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.489147 12373 raft_consensus.cc:515] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.491442 12373 leader_election.cc:290] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:57.492887 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:57.493513 12224 raft_consensus.cc:3055] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:57.493886 12224 raft_consensus.cc:740] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:57.493986 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4746e573b2142a09b3bc863b77051bd" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:57.494532 12300 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:57.494824 12224 consensus_queue.cc:260] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.495931 12570 raft_consensus.cc:993] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: : Instructing follower bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:57.496122 12224 raft_consensus.cc:3060] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:57.496539 12569 raft_consensus.cc:1081] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Signalling peer bc677a5b14684d43b945885131fe0922 to start an election
I20260504 19:59:57.496340 12121 catalog_manager.cc:5671] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d reported cstate change: term changed from 3 to 4, leader changed from 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1) to f9a4e75173b54be181f23568394a3a8d (127.11.68.2). New cstate: current_term: 4 leader_uuid: "f9a4e75173b54be181f23568394a3a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:57.498224 12372 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289"
dest_uuid: "bc677a5b14684d43b945885131fe0922"
from {username='slave'} at 127.0.0.1:47772
I20260504 19:59:57.498800 12372 raft_consensus.cc:493] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:57.499145 12372 raft_consensus.cc:3060] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:57.501471 12300 raft_consensus.cc:2468] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 3.
I20260504 19:59:57.502841 12340 leader_election.cc:304] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:57.503813 12549 raft_consensus.cc:2804] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:57.504580 12224 raft_consensus.cc:2468] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 3.
I20260504 19:59:57.506907 12372 raft_consensus.cc:515] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:57.508371 12372 leader_election.cc:290] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 2 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 19:59:57.509745 12300 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
I20260504 19:59:57.509748 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24d5d169f54c4b198069cf44a83e1289" candidate_uuid: "bc677a5b14684d43b945885131fe0922" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:57.510599 12300 raft_consensus.cc:3060] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:57.510684 12224 raft_consensus.cc:3055] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 19:59:57.511122 12224 raft_consensus.cc:740] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:57.512092 12224 consensus_queue.cc:260] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:57.513561 12224 raft_consensus.cc:3060] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Advancing to term 2
I20260504 19:59:57.516971 12549 raft_consensus.cc:697] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:57.517283 12300 raft_consensus.cc:2468] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 2.
I20260504 19:59:57.518584 12340 leader_election.cc:304] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:57.518903 12549 consensus_queue.cc:237] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.519615 12574 raft_consensus.cc:2804] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 19:59:57.522105 12224 raft_consensus.cc:2468] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc677a5b14684d43b945885131fe0922 in term 2.
I20260504 19:59:57.524279 12574 raft_consensus.cc:697] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 2 LEADER]: Becoming Leader. State: Replica: bc677a5b14684d43b945885131fe0922, State: Running, Role: LEADER
I20260504 19:59:57.525866 12574 consensus_queue.cc:237] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } }
I20260504 19:59:57.531679 12122 catalog_manager.cc:5671] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 2 to 3, leader changed from 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1) to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } }
I20260504 19:59:57.544752 12122 catalog_manager.cc:5671] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 reported cstate change: term changed from 1 to 2, leader changed from 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1) to bc677a5b14684d43b945885131fe0922 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "bc677a5b14684d43b945885131fe0922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:57.630576 12441 raft_consensus.cc:993] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: : Instructing follower f9a4e75173b54be181f23568394a3a8d to start an election
I20260504 19:59:57.631039 12570 raft_consensus.cc:1081] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Signalling peer f9a4e75173b54be181f23568394a3a8d to start an election
I20260504 19:59:57.632432 12300 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952"
dest_uuid: "f9a4e75173b54be181f23568394a3a8d"
from {username='slave'} at 127.0.0.1:55378
I20260504 19:59:57.632893 12300 raft_consensus.cc:493] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 19:59:57.633183 12300 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:57.637562 12300 raft_consensus.cc:515] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.639008 12300 leader_election.cc:290] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 3 election: Requested vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), bc677a5b14684d43b945885131fe0922 (127.11.68.3:38647)
I20260504 19:59:57.640008 12224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094"
I20260504 19:59:57.640019 12373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55a11f407d464fbe8753eed9f00ef952" candidate_uuid: "f9a4e75173b54be181f23568394a3a8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bc677a5b14684d43b945885131fe0922"
I20260504 19:59:57.640767 12224 raft_consensus.cc:3055] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 19:59:57.640833 12373 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:57.641238 12224 raft_consensus.cc:740] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, State: Running, Role: LEADER
I20260504 19:59:57.642026 12224 consensus_queue.cc:260] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.643338 12224 raft_consensus.cc:3060] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Advancing to term 3
I20260504 19:59:57.647838 12373 raft_consensus.cc:2468] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 3.
I20260504 19:59:57.649008 12264 leader_election.cc:304] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc677a5b14684d43b945885131fe0922, f9a4e75173b54be181f23568394a3a8d; no voters:
I20260504 19:59:57.649780 12542 raft_consensus.cc:2804] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Leader election won for term 3
I20260504 19:59:57.650035 12224 raft_consensus.cc:2468] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4e75173b54be181f23568394a3a8d in term 3.
I20260504 19:59:57.650318 12542 raft_consensus.cc:697] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 3 LEADER]: Becoming Leader. State: Replica: f9a4e75173b54be181f23568394a3a8d, State: Running, Role: LEADER
I20260504 19:59:57.651253 12542 consensus_queue.cc:237] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 19:59:57.658354 12122 catalog_manager.cc:5671] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d reported cstate change: term changed from 2 to 3, leader changed from 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1) to f9a4e75173b54be181f23568394a3a8d (127.11.68.2). New cstate: current_term: 3 leader_uuid: "f9a4e75173b54be181f23568394a3a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } health_report { overall_health: UNKNOWN } } }
I20260504 19:59:57.925657 12224 raft_consensus.cc:1275] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:57.926993 12574 consensus_queue.cc:1048] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:57.941742 12300 raft_consensus.cc:1275] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:57.942967 12549 consensus_queue.cc:1048] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:57.950444 12299 raft_consensus.cc:1275] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:57.951586 12549 consensus_queue.cc:1048] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:57.960753 12224 raft_consensus.cc:1275] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Refusing update from remote peer bc677a5b14684d43b945885131fe0922: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 19:59:57.962981 12574 consensus_queue.cc:1048] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 19:59:57.965915 12373 raft_consensus.cc:1275] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Refusing update from remote peer f9a4e75173b54be181f23568394a3a8d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:57.967572 12542 consensus_queue.cc:1048] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 19:59:57.980342 12224 raft_consensus.cc:1275] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Refusing update from remote peer f9a4e75173b54be181f23568394a3a8d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 19:59:57.982091 12571 consensus_queue.cc:1048] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 19:59:58.074347 12224 raft_consensus.cc:1275] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Refusing update from remote peer f9a4e75173b54be181f23568394a3a8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:58.075567 12571 consensus_queue.cc:1048] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:58.083894 12373 raft_consensus.cc:1275] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Refusing update from remote peer f9a4e75173b54be181f23568394a3a8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 19:59:58.085151 12542 consensus_queue.cc:1048] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 19:59:58.426578 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:58.432505 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:58.433115 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 19:59:59.434295 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 19:59:59.440773 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 19:59:59.441248 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:00.442277 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:00.447604 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 20:00:00.448091 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:01.449100 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:01.455796 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 20:00:01.456272 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:02.457219 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:02.463961 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 20:00:02.464416 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:03.465384 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:03.470935 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 20:00:03.471395 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:04.472441 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:04.478363 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 20:00:04.478825 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:05.480144 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:05.490727 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 20:00:05.491186 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:07.177158 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:07.182634 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260504 20:00:07.183171 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260504 20:00:07.703881 12321 debug-util.cc:398] Leaking SignalData structure 0x7b08000fb3c0 after lost signal to thread 11539
W20260504 20:00:07.704730 12321 debug-util.cc:398] Leaking SignalData structure 0x7b08000f8a60 after lost signal to thread 12157
W20260504 20:00:07.705555 12321 debug-util.cc:398] Leaking SignalData structure 0x7b08000a32e0 after lost signal to thread 12249
W20260504 20:00:07.706588 12321 debug-util.cc:398] Leaking SignalData structure 0x7b0800114440 after lost signal to thread 12324
W20260504 20:00:07.707413 12321 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba4c0 after lost signal to thread 12398
I20260504 20:00:08.223380 11536 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20260504 20:00:08.224074 11536 tablet_server.cc:179] TabletServer@127.11.68.1:0 shutting down...
W20260504 20:00:08.247624 12340 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 17 similar messages]
W20260504 20:00:08.255446 12340 consensus_peers.cc:597] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 -> Peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Couldn't send request to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:08.266865 11536 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260504 20:00:08.267940 11536 tablet_replica.cc:333] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.269134 11536 raft_consensus.cc:2243] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 LEADER]: Raft consensus shutting down.
I20260504 20:00:08.270569 11536 raft_consensus.cc:2272] T 734e7619181643f7a33111df773337b0 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.272977 11536 tablet_replica.cc:333] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.273551 11536 raft_consensus.cc:2243] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.274109 11536 raft_consensus.cc:2272] T 24d5d169f54c4b198069cf44a83e1289 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.276245 11536 tablet_replica.cc:333] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.276893 11536 raft_consensus.cc:2243] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.277449 11536 raft_consensus.cc:2272] T 0bd1d29b2cc94b5390cc815b5287f426 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.279387 11536 tablet_replica.cc:333] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.280047 11536 raft_consensus.cc:2243] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 LEADER]: Raft consensus shutting down.
I20260504 20:00:08.280941 11536 raft_consensus.cc:2272] T a2fd6cd6b68b47f999acb5ea52fde71e P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.282866 11536 tablet_replica.cc:333] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.283396 11536 raft_consensus.cc:2243] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.283883 11536 raft_consensus.cc:2272] T 55a11f407d464fbe8753eed9f00ef952 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.285794 11536 tablet_replica.cc:333] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.286325 11536 raft_consensus.cc:2243] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 LEADER]: Raft consensus shutting down.
I20260504 20:00:08.287209 11536 raft_consensus.cc:2272] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.289286 11536 tablet_replica.cc:333] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.289862 11536 raft_consensus.cc:2243] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 LEADER]: Raft consensus shutting down.
I20260504 20:00:08.290720 11536 raft_consensus.cc:2272] T cdaee4dfea3548dcb8579e74423edb9c P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.292619 11536 tablet_replica.cc:333] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.293115 11536 raft_consensus.cc:2243] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.293663 11536 raft_consensus.cc:2272] T 27d2807b93ae4423aeeeae7b5334de70 P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.295919 11536 tablet_replica.cc:333] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
I20260504 20:00:08.296542 11536 raft_consensus.cc:2243] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.297266 11536 raft_consensus.cc:2272] T 06a32550992a4221907f6d6a1bd147dd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.300036 11536 tablet_replica.cc:333] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094: stopping tablet replica
W20260504 20:00:08.300124 12266 consensus_peers.cc:597] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d -> Peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Couldn't send request to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:08.300601 11536 raft_consensus.cc:2243] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.301144 11536 raft_consensus.cc:2272] T d4746e573b2142a09b3bc863b77051bd P 7f2b1ad3a61b46e98ebd6aa6ac8b4094 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.335183 11536 tablet_server.cc:196] TabletServer@127.11.68.1:0 shutdown complete.
I20260504 20:00:08.370170 11536 tablet_server.cc:179] TabletServer@127.11.68.2:0 shutting down...
W20260504 20:00:08.387015 12340 consensus_peers.cc:597] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 -> Peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Couldn't send request to peer f9a4e75173b54be181f23568394a3a8d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260504 20:00:08.401870 12266 consensus_peers.cc:597] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d -> Peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Couldn't send request to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:08.413765 11536 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260504 20:00:08.414528 11536 tablet_replica.cc:333] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.415385 11536 raft_consensus.cc:2243] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.415906 11536 raft_consensus.cc:2272] T d4746e573b2142a09b3bc863b77051bd P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.418156 11536 tablet_replica.cc:333] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.418798 11536 raft_consensus.cc:2243] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.419370 11536 raft_consensus.cc:2272] T a2fd6cd6b68b47f999acb5ea52fde71e P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.421162 11536 tablet_replica.cc:333] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.421772 11536 raft_consensus.cc:2243] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 4 LEADER]: Raft consensus shutting down.
I20260504 20:00:08.422801 11536 raft_consensus.cc:2272] T 0bd1d29b2cc94b5390cc815b5287f426 P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.425530 11536 tablet_replica.cc:333] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.426288 11536 raft_consensus.cc:2243] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.426844 11536 raft_consensus.cc:2272] T 24d5d169f54c4b198069cf44a83e1289 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260504 20:00:08.427806 12266 consensus_peers.cc:597] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d -> Peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Couldn't send request to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:08.429167 11536 tablet_replica.cc:333] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.429740 11536 raft_consensus.cc:2243] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.430339 11536 raft_consensus.cc:2272] T cdaee4dfea3548dcb8579e74423edb9c P f9a4e75173b54be181f23568394a3a8d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.432277 11536 tablet_replica.cc:333] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.432874 11536 raft_consensus.cc:2243] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 3 LEADER]: Raft consensus shutting down.
I20260504 20:00:08.433990 11536 raft_consensus.cc:2272] T 55a11f407d464fbe8753eed9f00ef952 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.436064 11536 tablet_replica.cc:333] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.436583 11536 raft_consensus.cc:2243] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.437055 11536 raft_consensus.cc:2272] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.438968 11536 tablet_replica.cc:333] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.439512 11536 raft_consensus.cc:2243] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.439961 11536 raft_consensus.cc:2272] T 734e7619181643f7a33111df773337b0 P f9a4e75173b54be181f23568394a3a8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.441973 11536 tablet_replica.cc:333] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.442479 11536 raft_consensus.cc:2243] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:08.442942 11536 raft_consensus.cc:2272] T 27d2807b93ae4423aeeeae7b5334de70 P f9a4e75173b54be181f23568394a3a8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:08.444716 11536 tablet_replica.cc:333] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d: stopping tablet replica
I20260504 20:00:08.445181 11536 raft_consensus.cc:2243] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 1 LEADER]: Raft consensus shutting down.
I20260504 20:00:08.446076 11536 raft_consensus.cc:2272] T 06a32550992a4221907f6d6a1bd147dd P f9a4e75173b54be181f23568394a3a8d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 20:00:08.473688 12340 consensus_peers.cc:597] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 -> Peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Couldn't send request to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 20:00:08.508862 12340 consensus_peers.cc:597] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 -> Peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Couldn't send request to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 20:00:08.767464 12340 consensus_peers.cc:597] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 -> Peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Couldn't send request to peer f9a4e75173b54be181f23568394a3a8d. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 20:00:08.855420 12340 consensus_peers.cc:597] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 -> Peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Couldn't send request to peer f9a4e75173b54be181f23568394a3a8d. Status: Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:09.312605 12609 raft_consensus.cc:493] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7f2b1ad3a61b46e98ebd6aa6ac8b4094)
I20260504 20:00:09.313299 12609 raft_consensus.cc:515] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 20:00:09.315896 12609 leader_election.cc:290] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
I20260504 20:00:09.319098 12609 raft_consensus.cc:493] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7f2b1ad3a61b46e98ebd6aa6ac8b4094)
I20260504 20:00:09.319720 12609 raft_consensus.cc:515] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 20:00:09.321915 12609 leader_election.cc:290] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
W20260504 20:00:09.322171 12340 leader_election.cc:336] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111)
W20260504 20:00:09.324462 12340 leader_election.cc:336] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111)
I20260504 20:00:09.324858 12340 leader_election.cc:304] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 20:00:09.325515 12609 raft_consensus.cc:2749] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:00:09.329473 12340 leader_election.cc:336] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111)
W20260504 20:00:09.330160 12340 leader_election.cc:336] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111)
I20260504 20:00:09.330451 12340 leader_election.cc:304] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 20:00:09.331118 12609 raft_consensus.cc:2749] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:00:09.422062 12609 raft_consensus.cc:493] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7f2b1ad3a61b46e98ebd6aa6ac8b4094)
I20260504 20:00:09.422780 12609 raft_consensus.cc:515] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 20:00:09.425068 12609 leader_election.cc:290] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
W20260504 20:00:09.429088 12340 leader_election.cc:336] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111)
W20260504 20:00:09.430011 12340 leader_election.cc:336] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111)
I20260504 20:00:09.430363 12340 leader_election.cc:304] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 20:00:09.430991 12609 raft_consensus.cc:2749] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:00:09.476061 11536 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 12321)
I20260504 20:00:09.608270 12609 raft_consensus.cc:493] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f2b1ad3a61b46e98ebd6aa6ac8b4094)
I20260504 20:00:09.608863 12609 raft_consensus.cc:515] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 20:00:09.610589 12609 leader_election.cc:290] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
W20260504 20:00:09.614703 12340 leader_election.cc:336] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111)
W20260504 20:00:09.615422 12340 leader_election.cc:336] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111)
I20260504 20:00:09.615824 12340 leader_election.cc:304] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 20:00:09.616581 12609 raft_consensus.cc:2749] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:09.767232 12609 raft_consensus.cc:493] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f9a4e75173b54be181f23568394a3a8d)
I20260504 20:00:09.767807 12609 raft_consensus.cc:515] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 20:00:09.769599 12609 leader_election.cc:290] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
W20260504 20:00:09.773237 12340 leader_election.cc:336] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111)
W20260504 20:00:09.773949 12340 leader_election.cc:336] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111)
I20260504 20:00:09.774281 12340 leader_election.cc:304] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 20:00:09.774854 12609 raft_consensus.cc:2749] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:00:09.836886 12609 raft_consensus.cc:493] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f9a4e75173b54be181f23568394a3a8d)
I20260504 20:00:09.837509 12609 raft_consensus.cc:515] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 20:00:09.839823 12609 leader_election.cc:290] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
W20260504 20:00:09.845177 12340 leader_election.cc:336] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111)
W20260504 20:00:09.847460 12340 leader_election.cc:336] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111)
I20260504 20:00:09.847838 12340 leader_election.cc:304] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 20:00:09.848441 12609 raft_consensus.cc:2749] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:00:09.892354 12609 raft_consensus.cc:493] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a4e75173b54be181f23568394a3a8d)
I20260504 20:00:09.892858 12609 raft_consensus.cc:515] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f2b1ad3a61b46e98ebd6aa6ac8b4094" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 35487 } } peers { permanent_uuid: "f9a4e75173b54be181f23568394a3a8d" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 45537 } } peers { permanent_uuid: "bc677a5b14684d43b945885131fe0922" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 38647 } }
I20260504 20:00:09.894973 12609 leader_election.cc:290] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487), f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537)
W20260504 20:00:09.898628 12340 leader_election.cc:336] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9a4e75173b54be181f23568394a3a8d (127.11.68.2:45537): Network error: Client connection negotiation failed: client connection to 127.11.68.2:45537: connect: Connection refused (error 111)
W20260504 20:00:09.899469 12340 leader_election.cc:336] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f2b1ad3a61b46e98ebd6aa6ac8b4094 (127.11.68.1:35487): Network error: Client connection negotiation failed: client connection to 127.11.68.1:35487: connect: Connection refused (error 111)
I20260504 20:00:09.899814 12340 leader_election.cc:304] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc677a5b14684d43b945885131fe0922; no voters: 7f2b1ad3a61b46e98ebd6aa6ac8b4094, f9a4e75173b54be181f23568394a3a8d
I20260504 20:00:09.900422 12609 raft_consensus.cc:2749] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:10.082191 11536 tablet_server.cc:196] TabletServer@127.11.68.2:0 shutdown complete.
I20260504 20:00:10.102658 11536 tablet_server.cc:179] TabletServer@127.11.68.3:0 shutting down...
I20260504 20:00:10.126936 11536 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260504 20:00:10.127651 11536 tablet_replica.cc:333] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.128324 11536 raft_consensus.cc:2243] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 2 LEADER]: Raft consensus shutting down.
I20260504 20:00:10.129158 11536 raft_consensus.cc:2272] T 27d2807b93ae4423aeeeae7b5334de70 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.131322 11536 tablet_replica.cc:333] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.131836 11536 raft_consensus.cc:2243] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:10.132282 11536 raft_consensus.cc:2272] T 06a32550992a4221907f6d6a1bd147dd P bc677a5b14684d43b945885131fe0922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.134142 11536 tablet_replica.cc:333] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.134671 11536 raft_consensus.cc:2243] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:10.135164 11536 raft_consensus.cc:2272] T 0bd1d29b2cc94b5390cc815b5287f426 P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.136914 11536 tablet_replica.cc:333] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.137491 11536 raft_consensus.cc:2243] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:10.137987 11536 raft_consensus.cc:2272] T a2fd6cd6b68b47f999acb5ea52fde71e P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.139725 11536 tablet_replica.cc:333] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.140247 11536 raft_consensus.cc:2243] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 3 LEADER]: Raft consensus shutting down.
I20260504 20:00:10.141010 11536 raft_consensus.cc:2272] T d4746e573b2142a09b3bc863b77051bd P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.142865 11536 tablet_replica.cc:333] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.143383 11536 raft_consensus.cc:2243] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:10.143841 11536 raft_consensus.cc:2272] T 734e7619181643f7a33111df773337b0 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.145522 11536 tablet_replica.cc:333] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.146080 11536 raft_consensus.cc:2243] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:10.146652 11536 raft_consensus.cc:2272] T 55a11f407d464fbe8753eed9f00ef952 P bc677a5b14684d43b945885131fe0922 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.148391 11536 tablet_replica.cc:333] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.148885 11536 raft_consensus.cc:2243] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:10.149315 11536 raft_consensus.cc:2272] T cdaee4dfea3548dcb8579e74423edb9c P bc677a5b14684d43b945885131fe0922 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.151136 11536 tablet_replica.cc:333] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.151593 11536 raft_consensus.cc:2243] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:10.152019 11536 raft_consensus.cc:2272] T 0bbfa5c44a6d4d61a6175fe1e4a62d53 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.153702 11536 tablet_replica.cc:333] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922: stopping tablet replica
I20260504 20:00:10.154112 11536 raft_consensus.cc:2243] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 2 LEADER]: Raft consensus shutting down.
I20260504 20:00:10.154788 11536 raft_consensus.cc:2272] T 24d5d169f54c4b198069cf44a83e1289 P bc677a5b14684d43b945885131fe0922 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.185739 11536 tablet_server.cc:196] TabletServer@127.11.68.3:0 shutdown complete.
I20260504 20:00:10.206210 11536 master.cc:562] Master@127.11.68.62:37821 shutting down...
I20260504 20:00:10.229125 11536 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 1 LEADER]: Raft consensus shutting down.
I20260504 20:00:10.229773 11536 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:10.230208 11536 tablet_replica.cc:333] T 00000000000000000000000000000000 P ca7ec6b6819b4afb9fde3ea796094e2d: stopping tablet replica
I20260504 20:00:10.250486 11536 master.cc:584] Master@127.11.68.62:37821 shutdown complete.
7f2b1ad3a61b46e98ebd6aa6ac8b4094 5
bc677a5b14684d43b945885131fe0922 1
f9a4e75173b54be181f23568394a3a8d 4
7f2b1ad3a61b46e98ebd6aa6ac8b4094 3
bc677a5b14684d43b945885131fe0922 3
f9a4e75173b54be181f23568394a3a8d 4
7f2b1ad3a61b46e98ebd6aa6ac8b4094 8
bc677a5b14684d43b945885131fe0922 1
f9a4e75173b54be181f23568394a3a8d 1
7f2b1ad3a61b46e98ebd6aa6ac8b4094 4
bc677a5b14684d43b945885131fe0922 3
f9a4e75173b54be181f23568394a3a8d 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (31057 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260504 20:00:10.291879 11536 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.68.62:41865
I20260504 20:00:10.293083 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 20:00:10.299031 12624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 20:00:10.299096 12623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 20:00:10.299728 12626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 20:00:10.301144 11536 server_base.cc:1061] running on GCE node
I20260504 20:00:10.302240 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 20:00:10.302430 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 20:00:10.302549 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924810302538 us; error 0 us; skew 500 ppm
I20260504 20:00:10.303043 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 20:00:10.306165 11536 webserver.cc:492] Webserver started at http://127.11.68.62:42489/ using document root <none> and password file <none>
I20260504 20:00:10.306651 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 20:00:10.306833 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 20:00:10.307071 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 20:00:10.308244 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/master-0-root/instance:
uuid: "109800bd8ea34eaea0c7308025ecde32"
format_stamp: "Formatted at 2026-05-04 20:00:10 on dist-test-slave-3x8x"
I20260504 20:00:10.312832 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260504 20:00:10.316418 12631 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.317286 11536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260504 20:00:10.317596 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/master-0-root
uuid: "109800bd8ea34eaea0c7308025ecde32"
format_stamp: "Formatted at 2026-05-04 20:00:10 on dist-test-slave-3x8x"
I20260504 20:00:10.317953 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 20:00:10.336351 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 20:00:10.337708 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 20:00:10.377786 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.62:41865
I20260504 20:00:10.377983 12682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.62:41865 every 8 connection(s)
I20260504 20:00:10.381937 12683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:10.392341 12683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32: Bootstrap starting.
I20260504 20:00:10.397258 12683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:10.401908 12683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32: No bootstrap required, opened a new log
I20260504 20:00:10.404456 12683 raft_consensus.cc:359] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109800bd8ea34eaea0c7308025ecde32" member_type: VOTER }
I20260504 20:00:10.404908 12683 raft_consensus.cc:385] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:10.405241 12683 raft_consensus.cc:740] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109800bd8ea34eaea0c7308025ecde32, State: Initialized, Role: FOLLOWER
I20260504 20:00:10.405941 12683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109800bd8ea34eaea0c7308025ecde32" member_type: VOTER }
I20260504 20:00:10.406450 12683 raft_consensus.cc:399] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260504 20:00:10.406741 12683 raft_consensus.cc:493] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260504 20:00:10.407056 12683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:10.411552 12683 raft_consensus.cc:515] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109800bd8ea34eaea0c7308025ecde32" member_type: VOTER }
I20260504 20:00:10.412148 12683 leader_election.cc:304] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 109800bd8ea34eaea0c7308025ecde32; no voters:
I20260504 20:00:10.413376 12683 leader_election.cc:290] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260504 20:00:10.413722 12686 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:10.415241 12686 raft_consensus.cc:697] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [term 1 LEADER]: Becoming Leader. State: Replica: 109800bd8ea34eaea0c7308025ecde32, State: Running, Role: LEADER
I20260504 20:00:10.416100 12686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109800bd8ea34eaea0c7308025ecde32" member_type: VOTER }
I20260504 20:00:10.416862 12683 sys_catalog.cc:565] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [sys.catalog]: configured and running, proceeding with master startup.
I20260504 20:00:10.421509 12688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 109800bd8ea34eaea0c7308025ecde32. Latest consensus state: current_term: 1 leader_uuid: "109800bd8ea34eaea0c7308025ecde32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109800bd8ea34eaea0c7308025ecde32" member_type: VOTER } }
I20260504 20:00:10.422370 12688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [sys.catalog]: This master's current role is: LEADER
I20260504 20:00:10.424059 12687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "109800bd8ea34eaea0c7308025ecde32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109800bd8ea34eaea0c7308025ecde32" member_type: VOTER } }
I20260504 20:00:10.424832 12687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32 [sys.catalog]: This master's current role is: LEADER
I20260504 20:00:10.428519 12695 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 20:00:10.433383 12695 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 20:00:10.434615 11536 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260504 20:00:10.442521 12695 catalog_manager.cc:1357] Generated new cluster ID: 8d60fa3833f54b9086a339914030de9b
I20260504 20:00:10.442831 12695 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 20:00:10.459375 12695 catalog_manager.cc:1380] Generated new certificate authority record
I20260504 20:00:10.460675 12695 catalog_manager.cc:1514] Loading token signing keys...
I20260504 20:00:10.473425 12695 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32: Generated new TSK 0
I20260504 20:00:10.474045 12695 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 20:00:10.501654 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 20:00:10.507916 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 20:00:10.509011 12705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 20:00:10.510550 12707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 20:00:10.511483 11536 server_base.cc:1061] running on GCE node
I20260504 20:00:10.512519 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 20:00:10.512720 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 20:00:10.512900 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924810512861 us; error 0 us; skew 500 ppm
I20260504 20:00:10.513439 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 20:00:10.515799 11536 webserver.cc:492] Webserver started at http://127.11.68.1:41063/ using document root <none> and password file <none>
I20260504 20:00:10.516302 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 20:00:10.516501 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 20:00:10.516788 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 20:00:10.517961 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/instance:
uuid: "82787d8e64d24ca885ca517aa8ed1be2"
format_stamp: "Formatted at 2026-05-04 20:00:10 on dist-test-slave-3x8x"
I20260504 20:00:10.522578 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260504 20:00:10.526014 12712 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.526765 11536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260504 20:00:10.527081 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root
uuid: "82787d8e64d24ca885ca517aa8ed1be2"
format_stamp: "Formatted at 2026-05-04 20:00:10 on dist-test-slave-3x8x"
I20260504 20:00:10.527469 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 20:00:10.551726 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 20:00:10.553143 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 20:00:10.554740 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 20:00:10.557176 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 20:00:10.557408 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.557665 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 20:00:10.557893 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260504 20:00:10.600448 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.1:44007
I20260504 20:00:10.600528 12774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.1:44007 every 8 connection(s)
I20260504 20:00:10.605683 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 20:00:10.613399 12779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 20:00:10.614132 12780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 20:00:10.617722 11536 server_base.cc:1061] running on GCE node
W20260504 20:00:10.618701 12782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 20:00:10.619647 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 20:00:10.619946 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 20:00:10.620170 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924810620152 us; error 0 us; skew 500 ppm
I20260504 20:00:10.620954 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 20:00:10.623937 11536 webserver.cc:492] Webserver started at http://127.11.68.2:34203/ using document root <none> and password file <none>
I20260504 20:00:10.624507 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 20:00:10.624689 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 20:00:10.625000 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 20:00:10.626324 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/instance:
uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
format_stamp: "Formatted at 2026-05-04 20:00:10 on dist-test-slave-3x8x"
I20260504 20:00:10.632248 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260504 20:00:10.632596 12775 heartbeater.cc:344] Connected to a master server at 127.11.68.62:41865
I20260504 20:00:10.632984 12775 heartbeater.cc:461] Registering TS with master...
I20260504 20:00:10.633760 12775 heartbeater.cc:507] Master 127.11.68.62:41865 requested a full tablet report, sending...
I20260504 20:00:10.636114 12648 ts_manager.cc:194] Registered new tserver with Master: 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:10.636667 12787 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.637575 11536 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260504 20:00:10.637945 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root
uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
format_stamp: "Formatted at 2026-05-04 20:00:10 on dist-test-slave-3x8x"
I20260504 20:00:10.638273 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 20:00:10.638465 12648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39316
I20260504 20:00:10.658129 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 20:00:10.659523 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 20:00:10.661206 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 20:00:10.663707 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 20:00:10.663959 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.664235 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 20:00:10.664419 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.707875 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.2:41415
I20260504 20:00:10.707979 12849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.2:41415 every 8 connection(s)
I20260504 20:00:10.712862 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 20:00:10.720793 12853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 20:00:10.722492 12854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 20:00:10.724723 11536 server_base.cc:1061] running on GCE node
I20260504 20:00:10.725450 12850 heartbeater.cc:344] Connected to a master server at 127.11.68.62:41865
I20260504 20:00:10.725929 12850 heartbeater.cc:461] Registering TS with master...
W20260504 20:00:10.725934 12857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 20:00:10.726958 12850 heartbeater.cc:507] Master 127.11.68.62:41865 requested a full tablet report, sending...
I20260504 20:00:10.727005 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 20:00:10.727370 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 20:00:10.727638 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924810727615 us; error 0 us; skew 500 ppm
I20260504 20:00:10.728358 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 20:00:10.729184 12648 ts_manager.cc:194] Registered new tserver with Master: 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:10.731139 12648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39332
I20260504 20:00:10.731273 11536 webserver.cc:492] Webserver started at http://127.11.68.3:37063/ using document root <none> and password file <none>
I20260504 20:00:10.731920 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 20:00:10.732182 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 20:00:10.732486 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 20:00:10.733603 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/instance:
uuid: "732c3560a688494196d22a1a48f89968"
format_stamp: "Formatted at 2026-05-04 20:00:10 on dist-test-slave-3x8x"
I20260504 20:00:10.738093 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260504 20:00:10.741469 12861 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.742309 11536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260504 20:00:10.742563 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root
uuid: "732c3560a688494196d22a1a48f89968"
format_stamp: "Formatted at 2026-05-04 20:00:10 on dist-test-slave-3x8x"
I20260504 20:00:10.742808 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 20:00:10.775359 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 20:00:10.776533 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 20:00:10.778007 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 20:00:10.780285 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 20:00:10.780468 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.780781 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 20:00:10.780948 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:10.824311 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.3:39151
I20260504 20:00:10.824404 12923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.3:39151 every 8 connection(s)
I20260504 20:00:10.838634 12924 heartbeater.cc:344] Connected to a master server at 127.11.68.62:41865
I20260504 20:00:10.839001 12924 heartbeater.cc:461] Registering TS with master...
I20260504 20:00:10.839687 12924 heartbeater.cc:507] Master 127.11.68.62:41865 requested a full tablet report, sending...
I20260504 20:00:10.842059 12648 ts_manager.cc:194] Registered new tserver with Master: 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:10.842444 11536 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014953942s
I20260504 20:00:10.843539 12648 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39338
I20260504 20:00:10.870702 12648 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39352:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 20:00:10.876811 12648 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 20:00:11.097427 12815 tablet_service.cc:1511] Processing CreateTablet for tablet 2c03c00d6aaa4086b0726c5dfc8fd3ff (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260504 20:00:11.099028 12815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c03c00d6aaa4086b0726c5dfc8fd3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.100239 12811 tablet_service.cc:1511] Processing CreateTablet for tablet 37778afa14f24e82ade2bdc5c14b8a10 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260504 20:00:11.101779 12811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37778afa14f24e82ade2bdc5c14b8a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.107327 12889 tablet_service.cc:1511] Processing CreateTablet for tablet 2c03c00d6aaa4086b0726c5dfc8fd3ff (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260504 20:00:11.108748 12889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c03c00d6aaa4086b0726c5dfc8fd3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.103045 12814 tablet_service.cc:1511] Processing CreateTablet for tablet 445dd93bcaf04bae88b84d4eb9eddb60 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260504 20:00:11.110642 12814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445dd93bcaf04bae88b84d4eb9eddb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.110988 12807 tablet_service.cc:1511] Processing CreateTablet for tablet 71d0fab1020d42788e3e7a0c70e27637 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260504 20:00:11.112363 12807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71d0fab1020d42788e3e7a0c70e27637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.104005 12812 tablet_service.cc:1511] Processing CreateTablet for tablet 1e115d71520b4beb9b2984dd586354e4 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260504 20:00:11.115398 12812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e115d71520b4beb9b2984dd586354e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.104235 12813 tablet_service.cc:1511] Processing CreateTablet for tablet 0f8ca9e82e464c93b50b298fc802e0d0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260504 20:00:11.104455 12809 tablet_service.cc:1511] Processing CreateTablet for tablet 1ebe61f64fd74c158829e717ae37a5b9 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260504 20:00:11.127460 12809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ebe61f64fd74c158829e717ae37a5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.103164 12810 tablet_service.cc:1511] Processing CreateTablet for tablet c03254a421c44bdf89f865e637b18b2a (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260504 20:00:11.130251 12810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03254a421c44bdf89f865e637b18b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.105937 12808 tablet_service.cc:1511] Processing CreateTablet for tablet 57fe021572534bbdbbaea4aa4a64017a (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260504 20:00:11.133522 12813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f8ca9e82e464c93b50b298fc802e0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.134965 12808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57fe021572534bbdbbaea4aa4a64017a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.157099 12806 tablet_service.cc:1511] Processing CreateTablet for tablet 6d6a540765d9427aa634550381fa5eb2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260504 20:00:11.158723 12806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6a540765d9427aa634550381fa5eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.180763 12806 tablet_service.cc:1511] Processing CreateTablet for tablet 7cd4fffdb6254bd49f3d6ef2f95406ce (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260504 20:00:11.181602 12811 tablet_service.cc:1511] Processing CreateTablet for tablet 83c11e4ac731424b899e837119487d11 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260504 20:00:11.183331 12807 tablet_service.cc:1511] Processing CreateTablet for tablet 85ec7d979c6d41db891f52c7e0bbcefe (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260504 20:00:11.185722 12939 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:11.188261 12814 tablet_service.cc:1511] Processing CreateTablet for tablet 7f39e53fa82443a2ba4847a0750ee84c (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260504 20:00:11.189787 12806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd4fffdb6254bd49f3d6ef2f95406ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.194593 12887 tablet_service.cc:1511] Processing CreateTablet for tablet 0f8ca9e82e464c93b50b298fc802e0d0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260504 20:00:11.213531 12887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f8ca9e82e464c93b50b298fc802e0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.196414 12886 tablet_service.cc:1511] Processing CreateTablet for tablet 1e115d71520b4beb9b2984dd586354e4 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260504 20:00:11.218294 12814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f39e53fa82443a2ba4847a0750ee84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.218714 12886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e115d71520b4beb9b2984dd586354e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.218943 12939 tablet_bootstrap.cc:654] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:11.220510 12807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ec7d979c6d41db891f52c7e0bbcefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.198277 12885 tablet_service.cc:1511] Processing CreateTablet for tablet 37778afa14f24e82ade2bdc5c14b8a10 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260504 20:00:11.222719 12885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37778afa14f24e82ade2bdc5c14b8a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.200069 12884 tablet_service.cc:1511] Processing CreateTablet for tablet c03254a421c44bdf89f865e637b18b2a (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260504 20:00:11.226605 12884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03254a421c44bdf89f865e637b18b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.201848 12883 tablet_service.cc:1511] Processing CreateTablet for tablet 1ebe61f64fd74c158829e717ae37a5b9 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260504 20:00:11.230278 12883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ebe61f64fd74c158829e717ae37a5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.203657 12882 tablet_service.cc:1511] Processing CreateTablet for tablet 57fe021572534bbdbbaea4aa4a64017a (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260504 20:00:11.234709 12882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57fe021572534bbdbbaea4aa4a64017a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.205453 12881 tablet_service.cc:1511] Processing CreateTablet for tablet 71d0fab1020d42788e3e7a0c70e27637 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260504 20:00:11.237916 12812 tablet_service.cc:1511] Processing CreateTablet for tablet 8fd38813d75c466e8f0502c36726a913 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260504 20:00:11.238698 12881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71d0fab1020d42788e3e7a0c70e27637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.239269 12812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fd38813d75c466e8f0502c36726a913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.207015 12888 tablet_service.cc:1511] Processing CreateTablet for tablet 445dd93bcaf04bae88b84d4eb9eddb60 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260504 20:00:11.209201 12880 tablet_service.cc:1511] Processing CreateTablet for tablet 6d6a540765d9427aa634550381fa5eb2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260504 20:00:11.210861 12815 tablet_service.cc:1511] Processing CreateTablet for tablet e44b757a449142a38ed04025e01cb4a5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260504 20:00:11.241914 12811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c11e4ac731424b899e837119487d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.242672 12815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44b757a449142a38ed04025e01cb4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.244990 12939 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:11.245494 12939 ts_tablet_manager.cc:1403] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.003s
I20260504 20:00:11.246505 12888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445dd93bcaf04bae88b84d4eb9eddb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.248574 12939 raft_consensus.cc:359] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:11.249342 12939 raft_consensus.cc:385] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:11.250720 12880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6a540765d9427aa634550381fa5eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.263883 12809 tablet_service.cc:1511] Processing CreateTablet for tablet bce60695d3db4c06ad8e0eec0419f79f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260504 20:00:11.265283 12809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce60695d3db4c06ad8e0eec0419f79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.249624 12939 raft_consensus.cc:740] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:11.266597 12939 consensus_queue.cc:260] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:11.274319 12740 tablet_service.cc:1511] Processing CreateTablet for tablet 2c03c00d6aaa4086b0726c5dfc8fd3ff (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260504 20:00:11.275760 12740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c03c00d6aaa4086b0726c5dfc8fd3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.279830 12808 tablet_service.cc:1511] Processing CreateTablet for tablet 927c932a914241598bc58bcb0d84c2d4 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260504 20:00:11.281374 12808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927c932a914241598bc58bcb0d84c2d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.281119 12811 tablet_service.cc:1511] Processing CreateTablet for tablet fd0471a7f3374ceab25bb10c891e18e8 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260504 20:00:11.281733 12734 tablet_service.cc:1511] Processing CreateTablet for tablet 1ebe61f64fd74c158829e717ae37a5b9 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260504 20:00:11.282887 12811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0471a7f3374ceab25bb10c891e18e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.283128 12734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ebe61f64fd74c158829e717ae37a5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.283383 12737 tablet_service.cc:1511] Processing CreateTablet for tablet 1e115d71520b4beb9b2984dd586354e4 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260504 20:00:11.284701 12737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e115d71520b4beb9b2984dd586354e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.287016 12738 tablet_service.cc:1511] Processing CreateTablet for tablet 0f8ca9e82e464c93b50b298fc802e0d0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260504 20:00:11.287016 12733 tablet_service.cc:1511] Processing CreateTablet for tablet 57fe021572534bbdbbaea4aa4a64017a (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260504 20:00:11.279959 12735 tablet_service.cc:1511] Processing CreateTablet for tablet c03254a421c44bdf89f865e637b18b2a (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260504 20:00:11.288507 12738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f8ca9e82e464c93b50b298fc802e0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.289505 12733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57fe021572534bbdbbaea4aa4a64017a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.290639 12736 tablet_service.cc:1511] Processing CreateTablet for tablet 37778afa14f24e82ade2bdc5c14b8a10 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260504 20:00:11.291985 12736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37778afa14f24e82ade2bdc5c14b8a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.294042 12735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c03254a421c44bdf89f865e637b18b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.294981 12739 tablet_service.cc:1511] Processing CreateTablet for tablet 445dd93bcaf04bae88b84d4eb9eddb60 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260504 20:00:11.296325 12739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445dd93bcaf04bae88b84d4eb9eddb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.299654 12732 tablet_service.cc:1511] Processing CreateTablet for tablet 71d0fab1020d42788e3e7a0c70e27637 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260504 20:00:11.301093 12732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71d0fab1020d42788e3e7a0c70e27637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.303714 12731 tablet_service.cc:1511] Processing CreateTablet for tablet 6d6a540765d9427aa634550381fa5eb2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260504 20:00:11.315940 12731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6a540765d9427aa634550381fa5eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.330905 12886 tablet_service.cc:1511] Processing CreateTablet for tablet 83c11e4ac731424b899e837119487d11 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260504 20:00:11.337350 12886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c11e4ac731424b899e837119487d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.366688 12813 tablet_service.cc:1511] Processing CreateTablet for tablet dfa79aedaea6499facdfc14a3e7467f8 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260504 20:00:11.368692 12815 tablet_service.cc:1511] Processing CreateTablet for tablet f84c093f7c744182b34146f8f1e644d0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260504 20:00:11.390517 12807 tablet_service.cc:1511] Processing CreateTablet for tablet fea6d3aca49e4b1c9545e2ddb5b8e55b (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260504 20:00:11.398324 12811 tablet_service.cc:1511] Processing CreateTablet for tablet 4472f9104c524735bcfeed8474933a75 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260504 20:00:11.394730 12889 tablet_service.cc:1511] Processing CreateTablet for tablet 7cd4fffdb6254bd49f3d6ef2f95406ce (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260504 20:00:11.395107 12810 tablet_service.cc:1511] Processing CreateTablet for tablet a87aaa2ccb5340acb2654b93a91d22b5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260504 20:00:11.405614 12807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea6d3aca49e4b1c9545e2ddb5b8e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.407853 12814 tablet_service.cc:1511] Processing CreateTablet for tablet 410644ff9c0449188275a6d49ecc0c2d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260504 20:00:11.410709 12811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4472f9104c524735bcfeed8474933a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.411494 12806 tablet_service.cc:1511] Processing CreateTablet for tablet 00b8418b3a8441ee845a0591a75a9f17 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260504 20:00:11.412832 12806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b8418b3a8441ee845a0591a75a9f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.414158 12813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa79aedaea6499facdfc14a3e7467f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.418571 12940 tablet_bootstrap.cc:492] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:11.418915 12939 ts_tablet_manager.cc:1434] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.173s user 0.018s sys 0.028s
I20260504 20:00:11.420951 12850 heartbeater.cc:499] Master 127.11.68.62:41865 was elected leader, sending a full tablet report...
I20260504 20:00:11.421145 12939 tablet_bootstrap.cc:492] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:11.419520 12884 tablet_service.cc:1511] Processing CreateTablet for tablet 7f39e53fa82443a2ba4847a0750ee84c (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260504 20:00:11.423337 12810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a87aaa2ccb5340acb2654b93a91d22b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.423408 12943 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:11.423952 12943 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:11.428802 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:11.431959 12808 tablet_service.cc:1511] Processing CreateTablet for tablet ddb27f3eaaf74a3f9028f57cd8d93c9d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260504 20:00:11.433450 12808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb27f3eaaf74a3f9028f57cd8d93c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.438623 12814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410644ff9c0449188275a6d49ecc0c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.439440 12815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84c093f7c744182b34146f8f1e644d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.442472 12882 tablet_service.cc:1511] Processing CreateTablet for tablet 85ec7d979c6d41db891f52c7e0bbcefe (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260504 20:00:11.447299 12889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd4fffdb6254bd49f3d6ef2f95406ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.453533 12885 tablet_service.cc:1511] Processing CreateTablet for tablet e44b757a449142a38ed04025e01cb4a5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260504 20:00:11.454267 12884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f39e53fa82443a2ba4847a0750ee84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.457719 12943 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:11.458170 12885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44b757a449142a38ed04025e01cb4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.459154 12887 tablet_service.cc:1511] Processing CreateTablet for tablet 8fd38813d75c466e8f0502c36726a913 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
W20260504 20:00:11.462611 12851 tablet.cc:2404] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 20:00:11.462601 12882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ec7d979c6d41db891f52c7e0bbcefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.466166 12887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fd38813d75c466e8f0502c36726a913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.468569 12888 tablet_service.cc:1511] Processing CreateTablet for tablet 927c932a914241598bc58bcb0d84c2d4 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260504 20:00:11.468655 12881 tablet_service.cc:1511] Processing CreateTablet for tablet fd0471a7f3374ceab25bb10c891e18e8 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260504 20:00:11.487579 12881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0471a7f3374ceab25bb10c891e18e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.471658 12812 tablet_service.cc:1511] Processing CreateTablet for tablet 0e8eba4dbb8d436f81062ca3bbf46788 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260504 20:00:11.490700 12812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8eba4dbb8d436f81062ca3bbf46788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.461845 12880 tablet_service.cc:1511] Processing CreateTablet for tablet bce60695d3db4c06ad8e0eec0419f79f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260504 20:00:11.491912 12880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce60695d3db4c06ad8e0eec0419f79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.478812 12735 tablet_service.cc:1511] Processing CreateTablet for tablet 7cd4fffdb6254bd49f3d6ef2f95406ce (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260504 20:00:11.492934 12735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cd4fffdb6254bd49f3d6ef2f95406ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.493680 12888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927c932a914241598bc58bcb0d84c2d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.480445 12883 tablet_service.cc:1511] Processing CreateTablet for tablet dfa79aedaea6499facdfc14a3e7467f8 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260504 20:00:11.482090 12736 tablet_service.cc:1511] Processing CreateTablet for tablet fea6d3aca49e4b1c9545e2ddb5b8e55b (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260504 20:00:11.483551 12939 tablet_bootstrap.cc:654] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:11.473364 12886 tablet_service.cc:1511] Processing CreateTablet for tablet fea6d3aca49e4b1c9545e2ddb5b8e55b (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260504 20:00:11.498169 12944 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:11.479516 12940 tablet_bootstrap.cc:654] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:11.514154 12809 tablet_service.cc:1511] Processing CreateTablet for tablet 054fe686af134ff7b72d53edb9fdb32c (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260504 20:00:11.516247 12883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa79aedaea6499facdfc14a3e7467f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.517493 12886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea6d3aca49e4b1c9545e2ddb5b8e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.519477 12736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea6d3aca49e4b1c9545e2ddb5b8e55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.525318 12809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054fe686af134ff7b72d53edb9fdb32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.526573 12738 tablet_service.cc:1511] Processing CreateTablet for tablet 83c11e4ac731424b899e837119487d11 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260504 20:00:11.528055 12738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c11e4ac731424b899e837119487d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.534225 12734 tablet_service.cc:1511] Processing CreateTablet for tablet 7f39e53fa82443a2ba4847a0750ee84c (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260504 20:00:11.535220 12737 tablet_service.cc:1511] Processing CreateTablet for tablet 8fd38813d75c466e8f0502c36726a913 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260504 20:00:11.536527 12806 tablet_service.cc:1511] Processing CreateTablet for tablet f00885c8870d4ef98c5c0f9299c458ea (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260504 20:00:11.544045 12806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00885c8870d4ef98c5c0f9299c458ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.542549 12734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f39e53fa82443a2ba4847a0750ee84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.501788 12807 tablet_service.cc:1511] Processing CreateTablet for tablet 17c7354d697140a58c517d98642057ed (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260504 20:00:11.537773 12739 tablet_service.cc:1511] Processing CreateTablet for tablet e44b757a449142a38ed04025e01cb4a5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260504 20:00:11.538581 12733 tablet_service.cc:1511] Processing CreateTablet for tablet 85ec7d979c6d41db891f52c7e0bbcefe (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260504 20:00:11.543150 12811 tablet_service.cc:1511] Processing CreateTablet for tablet 7a80bf1f95a74ce09071a550fe15996f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260504 20:00:11.576650 12807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c7354d697140a58c517d98642057ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.582389 12740 tablet_service.cc:1511] Processing CreateTablet for tablet bce60695d3db4c06ad8e0eec0419f79f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260504 20:00:11.584375 12732 tablet_service.cc:1511] Processing CreateTablet for tablet 927c932a914241598bc58bcb0d84c2d4 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260504 20:00:11.572911 12737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fd38813d75c466e8f0502c36726a913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.582653 12810 tablet_service.cc:1511] Processing CreateTablet for tablet fa35f0107b5c4a97b13a1057c9b2bca2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260504 20:00:11.607232 12733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ec7d979c6d41db891f52c7e0bbcefe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.607823 12810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa35f0107b5c4a97b13a1057c9b2bca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.609318 12740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce60695d3db4c06ad8e0eec0419f79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.610129 12739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44b757a449142a38ed04025e01cb4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.588100 12813 tablet_service.cc:1511] Processing CreateTablet for tablet ae27dbfd0ab84c9d8c95896c0c4cf502 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260504 20:00:11.588685 12735 tablet_service.cc:1511] Processing CreateTablet for tablet fd0471a7f3374ceab25bb10c891e18e8 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260504 20:00:11.611840 12735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0471a7f3374ceab25bb10c891e18e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.612588 12732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 927c932a914241598bc58bcb0d84c2d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.590775 12882 tablet_service.cc:1511] Processing CreateTablet for tablet f84c093f7c744182b34146f8f1e644d0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260504 20:00:11.592443 12731 tablet_service.cc:1511] Processing CreateTablet for tablet dfa79aedaea6499facdfc14a3e7467f8 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260504 20:00:11.604205 12812 tablet_service.cc:1511] Processing CreateTablet for tablet 04d76f164cad4ed0bc8e39ce2403623f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260504 20:00:11.614441 12812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d76f164cad4ed0bc8e39ce2403623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.615406 12813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae27dbfd0ab84c9d8c95896c0c4cf502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.616047 12731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa79aedaea6499facdfc14a3e7467f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.629220 12881 tablet_service.cc:1511] Processing CreateTablet for tablet 4472f9104c524735bcfeed8474933a75 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260504 20:00:11.640570 12811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a80bf1f95a74ce09071a550fe15996f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.641449 12775 heartbeater.cc:499] Master 127.11.68.62:41865 was elected leader, sending a full tablet report...
I20260504 20:00:11.662756 12882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84c093f7c744182b34146f8f1e644d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.665288 12881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4472f9104c524735bcfeed8474933a75. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 20:00:11.672818 12635 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 81456.3us
I20260504 20:00:11.681222 12808 tablet_service.cc:1511] Processing CreateTablet for tablet d083393cd9f34e0b9d827ec5efd931eb (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260504 20:00:11.690234 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:11.703495 12791 leader_election.cc:343] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:11.705397 12885 tablet_service.cc:1511] Processing CreateTablet for tablet 00b8418b3a8441ee845a0591a75a9f17 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260504 20:00:11.708254 12880 tablet_service.cc:1511] Processing CreateTablet for tablet ddb27f3eaaf74a3f9028f57cd8d93c9d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260504 20:00:11.677673 12884 tablet_service.cc:1511] Processing CreateTablet for tablet 410644ff9c0449188275a6d49ecc0c2d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260504 20:00:11.710006 12814 tablet_service.cc:1511] Processing CreateTablet for tablet 756858b25f1f4125bf837cb8379e3b02 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260504 20:00:11.711618 12814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756858b25f1f4125bf837cb8379e3b02. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 20:00:11.715472 12633 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 121717us
I20260504 20:00:11.681443 12889 tablet_service.cc:1511] Processing CreateTablet for tablet 054fe686af134ff7b72d53edb9fdb32c (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260504 20:00:11.729776 12944 tablet_bootstrap.cc:654] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:11.756776 12808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d083393cd9f34e0b9d827ec5efd931eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.770666 12734 tablet_service.cc:1511] Processing CreateTablet for tablet f84c093f7c744182b34146f8f1e644d0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260504 20:00:11.772469 12734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84c093f7c744182b34146f8f1e644d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.777979 12731 tablet_service.cc:1511] Processing CreateTablet for tablet 4472f9104c524735bcfeed8474933a75 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260504 20:00:11.779826 12731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4472f9104c524735bcfeed8474933a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.782578 12811 tablet_service.cc:1511] Processing CreateTablet for tablet fe81ef9fbbe44f89a19e923171496fa5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260504 20:00:11.786859 12809 tablet_service.cc:1511] Processing CreateTablet for tablet 7e652e6edc2c4497b0f7f50dc79598f2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260504 20:00:11.795758 12887 tablet_service.cc:1511] Processing CreateTablet for tablet 0e8eba4dbb8d436f81062ca3bbf46788 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260504 20:00:11.808153 12811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe81ef9fbbe44f89a19e923171496fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.810674 12809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e652e6edc2c4497b0f7f50dc79598f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.810982 12886 tablet_service.cc:1511] Processing CreateTablet for tablet 17c7354d697140a58c517d98642057ed (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260504 20:00:11.784420 12884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410644ff9c0449188275a6d49ecc0c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.661371 12883 tablet_service.cc:1511] Processing CreateTablet for tablet a87aaa2ccb5340acb2654b93a91d22b5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260504 20:00:11.834738 12883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a87aaa2ccb5340acb2654b93a91d22b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.799665 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:11.846566 12924 heartbeater.cc:499] Master 127.11.68.62:41865 was elected leader, sending a full tablet report...
W20260504 20:00:11.875294 12791 leader_election.cc:343] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:11.875787 12791 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:11.876955 12943 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:11.892920 12738 tablet_service.cc:1511] Processing CreateTablet for tablet 410644ff9c0449188275a6d49ecc0c2d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260504 20:00:11.896811 12815 tablet_service.cc:1511] Processing CreateTablet for tablet 6673c1682edf457ba633ba714a47587f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260504 20:00:11.803699 12733 tablet_service.cc:1511] Processing CreateTablet for tablet a87aaa2ccb5340acb2654b93a91d22b5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260504 20:00:11.898676 12886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c7354d697140a58c517d98642057ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.865000 12736 tablet_service.cc:1511] Processing CreateTablet for tablet ae27dbfd0ab84c9d8c95896c0c4cf502 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260504 20:00:11.908262 12885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b8418b3a8441ee845a0591a75a9f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.909039 12944 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:11.909544 12944 ts_tablet_manager.cc:1403] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.412s user 0.040s sys 0.012s
I20260504 20:00:11.910228 12880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb27f3eaaf74a3f9028f57cd8d93c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.912479 12944 raft_consensus.cc:359] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:11.913214 12944 raft_consensus.cc:385] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:11.913496 12944 raft_consensus.cc:740] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:11.914165 12887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8eba4dbb8d436f81062ca3bbf46788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.914233 12944 consensus_queue.cc:260] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:11.917152 12939 tablet_bootstrap.cc:492] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:11.917692 12939 ts_tablet_manager.cc:1403] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.497s user 0.058s sys 0.083s
I20260504 20:00:11.930596 12738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410644ff9c0449188275a6d49ecc0c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.931303 12815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6673c1682edf457ba633ba714a47587f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.942013 12812 tablet_service.cc:1511] Processing CreateTablet for tablet c9a8aee69ec8499895f3ab719968d204 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260504 20:00:11.943465 12812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a8aee69ec8499895f3ab719968d204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.946466 12940 tablet_bootstrap.cc:492] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:11.947095 12814 tablet_service.cc:1511] Processing CreateTablet for tablet faf44d4cfb914599a82b50e6d195ecbd (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260504 20:00:11.954741 12889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054fe686af134ff7b72d53edb9fdb32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.973177 12733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a87aaa2ccb5340acb2654b93a91d22b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.973295 12737 tablet_service.cc:1511] Processing CreateTablet for tablet 00b8418b3a8441ee845a0591a75a9f17 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260504 20:00:11.974642 12737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b8418b3a8441ee845a0591a75a9f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.975221 12740 tablet_service.cc:1511] Processing CreateTablet for tablet 0e8eba4dbb8d436f81062ca3bbf46788 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260504 20:00:11.976500 12740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8eba4dbb8d436f81062ca3bbf46788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.978415 12814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf44d4cfb914599a82b50e6d195ecbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.979310 12739 tablet_service.cc:1511] Processing CreateTablet for tablet ddb27f3eaaf74a3f9028f57cd8d93c9d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260504 20:00:11.980751 12739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb27f3eaaf74a3f9028f57cd8d93c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.980970 12888 tablet_service.cc:1511] Processing CreateTablet for tablet fa35f0107b5c4a97b13a1057c9b2bca2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260504 20:00:11.987720 12940 ts_tablet_manager.cc:1403] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.569s user 0.032s sys 0.077s
I20260504 20:00:11.983098 12736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae27dbfd0ab84c9d8c95896c0c4cf502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:11.949105 12813 tablet_service.cc:1511] Processing CreateTablet for tablet 47a6e494e974412e9a7112344021d967 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260504 20:00:11.992942 12882 tablet_service.cc:1511] Processing CreateTablet for tablet 7a80bf1f95a74ce09071a550fe15996f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260504 20:00:12.029897 12888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa35f0107b5c4a97b13a1057c9b2bca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.039201 12882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a80bf1f95a74ce09071a550fe15996f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.034889 12944 ts_tablet_manager.cc:1434] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.125s user 0.021s sys 0.012s
I20260504 20:00:12.039094 12939 raft_consensus.cc:359] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.039968 12939 raft_consensus.cc:385] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:12.040284 12939 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.041007 12939 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.047117 12813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a6e494e974412e9a7112344021d967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.049662 12944 tablet_bootstrap.cc:492] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:12.045465 12884 tablet_service.cc:1511] Processing CreateTablet for tablet f00885c8870d4ef98c5c0f9299c458ea (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260504 20:00:12.051195 12884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00885c8870d4ef98c5c0f9299c458ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.052148 12735 tablet_service.cc:1511] Processing CreateTablet for tablet 17c7354d697140a58c517d98642057ed (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260504 20:00:12.057725 12806 tablet_service.cc:1511] Processing CreateTablet for tablet 7ddedcb68670423f9dbf822898141e0f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260504 20:00:12.058142 12950 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:12.063341 12950 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.066452 12881 tablet_service.cc:1511] Processing CreateTablet for tablet 04d76f164cad4ed0bc8e39ce2403623f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260504 20:00:12.066919 12807 tablet_service.cc:1511] Processing CreateTablet for tablet 5409cb132bc14d1cafaa29b22ced9dc9 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260504 20:00:12.068022 12881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d76f164cad4ed0bc8e39ce2403623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.072892 12940 raft_consensus.cc:359] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:12.073697 12940 raft_consensus.cc:385] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:12.074031 12940 raft_consensus.cc:740] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.074717 12732 tablet_service.cc:1511] Processing CreateTablet for tablet 054fe686af134ff7b72d53edb9fdb32c (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260504 20:00:12.080176 12887 tablet_service.cc:1511] Processing CreateTablet for tablet ae27dbfd0ab84c9d8c95896c0c4cf502 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260504 20:00:12.081530 12887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae27dbfd0ab84c9d8c95896c0c4cf502. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 20:00:12.090035 12635 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 43912.2us
I20260504 20:00:12.060657 12735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c7354d697140a58c517d98642057ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.097995 12886 tablet_service.cc:1511] Processing CreateTablet for tablet 756858b25f1f4125bf837cb8379e3b02 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260504 20:00:12.098052 12885 tablet_service.cc:1511] Processing CreateTablet for tablet d083393cd9f34e0b9d827ec5efd931eb (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260504 20:00:12.093389 12950 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:12.098381 12808 tablet_service.cc:1511] Processing CreateTablet for tablet af25f180a18a4e79a93b35bb0a2e55f0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260504 20:00:12.085430 12734 tablet_service.cc:1511] Processing CreateTablet for tablet f00885c8870d4ef98c5c0f9299c458ea (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260504 20:00:12.122017 12732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054fe686af134ff7b72d53edb9fdb32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.124634 12880 tablet_service.cc:1511] Processing CreateTablet for tablet fe81ef9fbbe44f89a19e923171496fa5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260504 20:00:12.112323 12886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756858b25f1f4125bf837cb8379e3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.139496 12883 tablet_service.cc:1511] Processing CreateTablet for tablet 7e652e6edc2c4497b0f7f50dc79598f2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260504 20:00:12.147671 12731 tablet_service.cc:1511] Processing CreateTablet for tablet 7a80bf1f95a74ce09071a550fe15996f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260504 20:00:12.152673 12883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e652e6edc2c4497b0f7f50dc79598f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.153899 12806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddedcb68670423f9dbf822898141e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.158010 12810 tablet_service.cc:1511] Processing CreateTablet for tablet 0832607bf7dd4d93a376269f0dd3dca1 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260504 20:00:12.159453 12810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0832607bf7dd4d93a376269f0dd3dca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.166363 12808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af25f180a18a4e79a93b35bb0a2e55f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.167307 12889 tablet_service.cc:1511] Processing CreateTablet for tablet 6673c1682edf457ba633ba714a47587f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260504 20:00:12.170961 12939 ts_tablet_manager.cc:1434] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.253s user 0.011s sys 0.002s
I20260504 20:00:12.177026 12939 tablet_bootstrap.cc:492] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:12.176931 12809 tablet_service.cc:1511] Processing CreateTablet for tablet 43dc05259beb4569a29aeb5327c9b3bc (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260504 20:00:12.074741 12940 consensus_queue.cc:260] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:12.179086 12885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d083393cd9f34e0b9d827ec5efd931eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.179370 12734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00885c8870d4ef98c5c0f9299c458ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.179719 12880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe81ef9fbbe44f89a19e923171496fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.183859 12731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a80bf1f95a74ce09071a550fe15996f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.187698 12740 tablet_service.cc:1511] Processing CreateTablet for tablet fa35f0107b5c4a97b13a1057c9b2bca2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260504 20:00:12.189030 12740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa35f0107b5c4a97b13a1057c9b2bca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.189932 12944 tablet_bootstrap.cc:700] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent opening tablet: real 0.100s user 0.001s sys 0.004s
I20260504 20:00:12.190611 12944 tablet_bootstrap.cc:654] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.179086 12807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5409cb132bc14d1cafaa29b22ced9dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.192714 12889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6673c1682edf457ba633ba714a47587f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.207405 12737 tablet_service.cc:1511] Processing CreateTablet for tablet d083393cd9f34e0b9d827ec5efd931eb (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260504 20:00:12.209337 12737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d083393cd9f34e0b9d827ec5efd931eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.231639 12809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43dc05259beb4569a29aeb5327c9b3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.232893 12943 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:12.233382 12943 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.235350 12943 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:12.235116 12812 tablet_service.cc:1511] Processing CreateTablet for tablet 91ff767ed7c24b97940a459421881055 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260504 20:00:12.236397 12812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ff767ed7c24b97940a459421881055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.237123 12884 tablet_service.cc:1511] Processing CreateTablet for tablet faf44d4cfb914599a82b50e6d195ecbd (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260504 20:00:12.239498 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:12.241451 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:12.244642 12791 leader_election.cc:343] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:12.245373 12791 leader_election.cc:343] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.245782 12791 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:12.246599 12943 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:12.249171 12733 tablet_service.cc:1511] Processing CreateTablet for tablet fe81ef9fbbe44f89a19e923171496fa5 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260504 20:00:12.249830 12888 tablet_service.cc:1511] Processing CreateTablet for tablet 47a6e494e974412e9a7112344021d967 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260504 20:00:12.251202 12814 tablet_service.cc:1511] Processing CreateTablet for tablet f2114545cfec4b19a5a3d1d6751f73da (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260504 20:00:12.262620 12811 tablet_service.cc:1511] Processing CreateTablet for tablet b1271e7bfcea48b8bd072293da65c55d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260504 20:00:12.266587 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:12.278414 12825 raft_consensus.cc:2468] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:12.279052 12814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2114545cfec4b19a5a3d1d6751f73da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.279910 12714 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:12.280686 12888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a6e494e974412e9a7112344021d967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.280864 12950 raft_consensus.cc:2804] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:12.281241 12950 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:12.281567 12950 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:12.276367 12736 tablet_service.cc:1511] Processing CreateTablet for tablet 04d76f164cad4ed0bc8e39ce2403623f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260504 20:00:12.271529 12815 tablet_service.cc:1511] Processing CreateTablet for tablet 86ea905fd808496c9b7d84a57c496cb3 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260504 20:00:12.271852 12939 tablet_bootstrap.cc:654] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.286412 12884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf44d4cfb914599a82b50e6d195ecbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.296600 12950 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.225313 12738 tablet_service.cc:1511] Processing CreateTablet for tablet 756858b25f1f4125bf837cb8379e3b02 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260504 20:00:12.286809 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:12.308372 12716 leader_election.cc:343] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.308642 12950 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:12.310416 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:00:12.312101 12716 leader_election.cc:343] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.312767 12944 tablet_bootstrap.cc:492] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:12.313383 12944 ts_tablet_manager.cc:1403] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.264s user 0.010s sys 0.011s
I20260504 20:00:12.314193 12733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe81ef9fbbe44f89a19e923171496fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.314837 12738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756858b25f1f4125bf837cb8379e3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.316637 12944 raft_consensus.cc:359] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.318747 12736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d76f164cad4ed0bc8e39ce2403623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.321092 12811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1271e7bfcea48b8bd072293da65c55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.320268 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:12.323174 12825 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:12.285575 12882 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 31) took 1013 ms. Trace:
I20260504 20:00:12.333825 12825 raft_consensus.cc:2468] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:12.335157 12714 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:12.336398 12950 raft_consensus.cc:2804] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:12.341078 12940 ts_tablet_manager.cc:1434] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.298s user 0.032s sys 0.023s
I20260504 20:00:12.342217 12940 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:12.344049 12731 tablet_service.cc:1511] Processing CreateTablet for tablet faf44d4cfb914599a82b50e6d195ecbd (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260504 20:00:12.345458 12731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf44d4cfb914599a82b50e6d195ecbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.346040 12740 tablet_service.cc:1511] Processing CreateTablet for tablet 7e652e6edc2c4497b0f7f50dc79598f2 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260504 20:00:12.347338 12740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e652e6edc2c4497b0f7f50dc79598f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.348829 12940 tablet_bootstrap.cc:654] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.350548 12939 tablet_bootstrap.cc:492] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:12.351074 12939 ts_tablet_manager.cc:1403] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.174s user 0.007s sys 0.009s
I20260504 20:00:12.354014 12954 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:12.354547 12954 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:12.357368 12954 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:12.325598 12944 raft_consensus.cc:385] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:12.361856 12944 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.360296 12739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 26) took 1011 ms. Trace:
I20260504 20:00:12.351315 12884 tablet_service.cc:1511] Processing CreateTablet for tablet 7ddedcb68670423f9dbf822898141e0f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260504 20:00:12.377275 12815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ea905fd808496c9b7d84a57c496cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.371183 12732 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 28) took 1023 ms. Trace:
I20260504 20:00:12.325960 12882 rpcz_store.cc:276] 0504 20:00:11.272556 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:11.991682 (+719126us) service_pool.cc:224] Handling call
0504 20:00:12.038737 (+ 47055us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.038832 (+ 95us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.203636 (+164804us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.285034 (+ 81398us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.285507 (+ 473us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.393272 12882 tablet_service.cc:1511] Processing CreateTablet for tablet 5409cb132bc14d1cafaa29b22ced9dc9 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260504 20:00:12.382320 12732 rpcz_store.cc:276] 0504 20:00:11.347799 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.041150 (+693351us) service_pool.cc:224] Handling call
0504 20:00:12.079573 (+ 38423us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.079673 (+ 100us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.232243 (+152570us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.353848 (+121605us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.370006 (+ 16158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.407356 12732 tablet_service.cc:1511] Processing CreateTablet for tablet 6673c1682edf457ba633ba714a47587f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260504 20:00:12.411808 12732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6673c1682edf457ba633ba714a47587f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.372398 12739 rpcz_store.cc:276] 0504 20:00:11.347237 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:11.973320 (+626083us) service_pool.cc:224] Handling call
0504 20:00:11.980291 (+ 6971us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:11.980401 (+ 110us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.121664 (+141263us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.276974 (+155310us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.358726 (+ 81752us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.417641 12739 tablet_service.cc:1511] Processing CreateTablet for tablet c9a8aee69ec8499895f3ab719968d204 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260504 20:00:12.418634 12807 tablet_service.cc:1511] Processing CreateTablet for tablet 431221deee84429b8261f87972c51218 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260504 20:00:12.418769 12885 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 35) took 1091 ms. Trace:
I20260504 20:00:12.419353 12885 rpcz_store.cc:276] 0504 20:00:11.327005 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.096987 (+769982us) service_pool.cc:224] Handling call
0504 20:00:12.099203 (+ 2216us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.099320 (+ 117us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.264747 (+165427us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.415747 (+151000us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.418689 (+ 2942us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5189}
I20260504 20:00:12.370396 12883 tablet_service.cc:1511] Processing CreateTablet for tablet c9a8aee69ec8499895f3ab719968d204 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260504 20:00:12.422197 12883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a8aee69ec8499895f3ab719968d204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.422535 12885 tablet_service.cc:1511] Processing CreateTablet for tablet af25f180a18a4e79a93b35bb0a2e55f0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260504 20:00:12.423818 12885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af25f180a18a4e79a93b35bb0a2e55f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.428355 12881 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 33) took 1118 ms. Trace:
I20260504 20:00:12.428884 12881 rpcz_store.cc:276] 0504 20:00:11.310153 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.065471 (+755318us) service_pool.cc:224] Handling call
0504 20:00:12.067374 (+ 1903us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.067505 (+ 131us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.217109 (+149604us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.427706 (+210597us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.428276 (+ 570us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.431828 12881 tablet_service.cc:1511] Processing CreateTablet for tablet 0832607bf7dd4d93a376269f0dd3dca1 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260504 20:00:12.433568 12881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0832607bf7dd4d93a376269f0dd3dca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.439359 12738 tablet_service.cc:1511] Processing CreateTablet for tablet 47a6e494e974412e9a7112344021d967 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260504 20:00:12.413169 12813 tablet_service.cc:1511] Processing CreateTablet for tablet aada4371a0e5404bb6b69e5a2f3b1164 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260504 20:00:12.441416 12887 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 34) took 1114 ms. Trace:
I20260504 20:00:12.485672 12887 rpcz_store.cc:276] 0504 20:00:11.326580 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.079193 (+752613us) service_pool.cc:224] Handling call
0504 20:00:12.081048 (+ 1855us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.081143 (+ 95us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.252860 (+171717us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.414273 (+161413us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.441349 (+ 27076us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.486740 12882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5409cb132bc14d1cafaa29b22ced9dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.488023 12734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 30) took 1139 ms. Trace:
I20260504 20:00:12.488492 12734 rpcz_store.cc:276] 0504 20:00:11.348357 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.084462 (+736105us) service_pool.cc:224] Handling call
0504 20:00:12.123993 (+ 39531us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.124166 (+ 173us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.252049 (+127883us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.427966 (+175917us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.487956 (+ 59990us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.489615 12735 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 29) took 1141 ms. Trace:
I20260504 20:00:12.490603 12734 tablet_service.cc:1511] Processing CreateTablet for tablet 7ddedcb68670423f9dbf822898141e0f (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260504 20:00:12.399365 12944 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.444108 12807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431221deee84429b8261f87972c51218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.490140 12735 rpcz_store.cc:276] 0504 20:00:11.348071 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.050931 (+702860us) service_pool.cc:224] Handling call
0504 20:00:12.053246 (+ 2315us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.053372 (+ 126us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.253482 (+200110us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.442134 (+188652us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.489533 (+ 47399us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.427984 12939 raft_consensus.cc:359] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.493036 12939 raft_consensus.cc:385] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:12.493458 12939 raft_consensus.cc:740] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.493569 12735 tablet_service.cc:1511] Processing CreateTablet for tablet 5409cb132bc14d1cafaa29b22ced9dc9 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260504 20:00:12.452339 12886 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 36) took 1124 ms. Trace:
I20260504 20:00:12.494565 12886 rpcz_store.cc:276] 0504 20:00:11.327423 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.097047 (+769624us) service_pool.cc:224] Handling call
0504 20:00:12.098948 (+ 1901us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.099038 (+ 90us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.188470 (+ 89432us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.393380 (+204910us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.451783 (+ 58403us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3cf2 00007f01627b9469 00007f016190b488 00007f01619d2033 00007f01619cddcf 00007f01619ce58d 00007f016197cac0 00007f015adf2bf4 00007f015adf2b82
0504 20:00:12.452254 (+ 471us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10927360}
I20260504 20:00:12.455716 12808 tablet_service.cc:1511] Processing CreateTablet for tablet 640695edb69f427ea7660c9522bb01ef (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260504 20:00:12.498835 12808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 640695edb69f427ea7660c9522bb01ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.470654 12811 tablet_service.cc:1511] Processing CreateTablet for tablet 4318d305513d467c8254d7c054ef993e (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260504 20:00:12.500945 12811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4318d305513d467c8254d7c054ef993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.497191 12886 tablet_service.cc:1511] Processing CreateTablet for tablet 91ff767ed7c24b97940a459421881055 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260504 20:00:12.488345 12887 tablet_service.cc:1511] Processing CreateTablet for tablet 43dc05259beb4569a29aeb5327c9b3bc (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260504 20:00:12.384918 12809 tablet_service.cc:1511] Processing CreateTablet for tablet 593a993b4ff341c9b98629e2c051ce00 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20260504 20:00:12.447248 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:12.488045 12884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddedcb68670423f9dbf822898141e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.523273 12950 raft_consensus.cc:697] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:12.529352 12886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ff767ed7c24b97940a459421881055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.531236 12736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 33) took 1181 ms. Trace:
I20260504 20:00:12.535151 12887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43dc05259beb4569a29aeb5327c9b3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.535735 12950 consensus_queue.cc:237] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.539443 12880 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 37) took 1211 ms. Trace:
I20260504 20:00:12.539969 12880 rpcz_store.cc:276] 0504 20:00:11.327826 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.123617 (+795791us) service_pool.cc:224] Handling call
0504 20:00:12.151596 (+ 27979us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.151790 (+ 194us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.296134 (+144344us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.538890 (+242756us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.539376 (+ 486us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68224}
I20260504 20:00:12.527299 12939 consensus_queue.cc:260] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.542804 12939 ts_tablet_manager.cc:1434] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.191s user 0.013s sys 0.000s
I20260504 20:00:12.545545 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:12.549868 12863 leader_election.cc:343] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.550896 12809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593a993b4ff341c9b98629e2c051ce00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.554499 12813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aada4371a0e5404bb6b69e5a2f3b1164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.556334 12733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 37) took 1164 ms. Trace:
I20260504 20:00:12.556861 12733 rpcz_store.cc:276] 0504 20:00:11.391921 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.248202 (+856281us) service_pool.cc:224] Handling call
0504 20:00:12.289969 (+ 41767us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.298164 (+ 8195us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.379658 (+ 81494us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.555707 (+176049us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.556264 (+ 557us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.562366 12806 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53094 (request call id 43) took 1108 ms. Trace:
I20260504 20:00:12.523940 12739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a8aee69ec8499895f3ab719968d204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.575978 12940 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:12.585075 12944 ts_tablet_manager.cc:1434] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.271s user 0.010s sys 0.007s
I20260504 20:00:12.585631 12940 ts_tablet_manager.cc:1403] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.244s user 0.011s sys 0.031s
I20260504 20:00:12.588652 12944 tablet_bootstrap.cc:492] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:12.594696 12889 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 39) took 1260 ms. Trace:
I20260504 20:00:12.594933 12733 tablet_service.cc:1511] Processing CreateTablet for tablet af25f180a18a4e79a93b35bb0a2e55f0 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260504 20:00:12.595229 12889 rpcz_store.cc:276] 0504 20:00:11.334298 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.166081 (+831783us) service_pool.cc:224] Handling call
0504 20:00:12.190145 (+ 24064us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.190613 (+ 468us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.435554 (+244941us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.594165 (+158611us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.594630 (+ 465us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.598040 12940 raft_consensus.cc:359] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.598846 12940 raft_consensus.cc:385] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:12.599177 12940 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.566200 12736 rpcz_store.cc:276] 0504 20:00:11.349320 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.189644 (+840324us) service_pool.cc:224] Handling call
0504 20:00:12.303432 (+113788us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.305779 (+ 2347us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.368633 (+ 62854us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.530691 (+162058us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.531171 (+ 480us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.600020 12940 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.567004 12806 rpcz_store.cc:276] 0504 20:00:11.454057 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.056470 (+602413us) service_pool.cc:224] Handling call
0504 20:00:12.093240 (+ 36770us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.100692 (+ 7452us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.276580 (+175888us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.528670 (+252090us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.542587 (+ 13917us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3cf2 00007f01627b9469 00007f016190b488 00007f01619d2033 00007f01619cddcf 00007f01619ce58d 00007f016197cac0 00007f015adf2bf4 00007f015adf2b82
0504 20:00:12.562294 (+ 19707us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12086144}
I20260504 20:00:12.602770 12814 tablet_service.cc:1511] Processing CreateTablet for tablet 36b69b3d13384774a38b1442833e6948 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260504 20:00:12.604554 12814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b69b3d13384774a38b1442833e6948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.605158 12736 tablet_service.cc:1511] Processing CreateTablet for tablet 0832607bf7dd4d93a376269f0dd3dca1 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260504 20:00:12.606073 12738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a6e494e974412e9a7112344021d967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.608850 12806 tablet_service.cc:1511] Processing CreateTablet for tablet e67d58ddf5464b6daea5f094d46a05b6 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260504 20:00:12.599090 12880 tablet_service.cc:1511] Processing CreateTablet for tablet f2114545cfec4b19a5a3d1d6751f73da (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260504 20:00:12.618391 12940 ts_tablet_manager.cc:1434] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260504 20:00:12.619943 12964 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:12.617499 12889 tablet_service.cc:1511] Processing CreateTablet for tablet b1271e7bfcea48b8bd072293da65c55d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260504 20:00:12.639279 12806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67d58ddf5464b6daea5f094d46a05b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.630676 12964 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.665122 12732 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 39) took 1272 ms. Trace:
I20260504 20:00:12.628756 12740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 38) took 1228 ms. Trace:
I20260504 20:00:12.632582 12888 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 42) took 1227 ms. Trace:
I20260504 20:00:12.635540 12883 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 40) took 1296 ms. Trace:
I20260504 20:00:12.643157 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:12.626408 12939 tablet_bootstrap.cc:492] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:12.628189 12886 tablet_service.cc:1511] Processing CreateTablet for tablet 86ea905fd808496c9b7d84a57c496cb3 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260504 20:00:12.675676 12865 leader_election.cc:343] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.676151 12865 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:12.677161 12956 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:12.651971 12648 catalog_manager.cc:5671] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:12.677830 12731 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 41) took 1275 ms. Trace:
I20260504 20:00:12.654664 12940 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:12.627720 12737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 35) took 1264 ms. Trace:
I20260504 20:00:12.678664 12737 rpcz_store.cc:276] 0504 20:00:11.381303 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.204233 (+822930us) service_pool.cc:224] Handling call
0504 20:00:12.208605 (+ 4372us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.208748 (+ 143us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.436807 (+228059us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.627209 (+190402us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.627657 (+ 448us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.680799 12737 tablet_service.cc:1511] Processing CreateTablet for tablet 43dc05259beb4569a29aeb5327c9b3bc (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260504 20:00:12.675359 12883 rpcz_store.cc:276] 0504 20:00:11.339434 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.177742 (+838308us) service_pool.cc:224] Handling call
0504 20:00:12.421542 (+243800us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.421668 (+ 126us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.564827 (+143159us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.635028 (+ 70201us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.635477 (+ 449us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.631496 12734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddedcb68670423f9dbf822898141e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.689538 12739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 40) took 1296 ms. Trace:
I20260504 20:00:12.690179 12939 tablet_bootstrap.cc:654] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.684782 12735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5409cb132bc14d1cafaa29b22ced9dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.640429 12884 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 43) took 1183 ms. Trace:
I20260504 20:00:12.669934 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:12.671005 12888 rpcz_store.cc:276] 0504 20:00:11.405331 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.248882 (+843551us) service_pool.cc:224] Handling call
0504 20:00:12.280049 (+ 31167us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619f0ef1 00007f01619ce183 00007f016197cac0 00007f015adf2bf4 00007f015adf2b82 00007f015adf2ab1
0504 20:00:12.280193 (+ 144us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.280282 (+ 89us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.398341 (+118059us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.632101 (+233760us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.632524 (+ 423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7472000}
I20260504 20:00:12.694085 12736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0832607bf7dd4d93a376269f0dd3dca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.693951 12884 rpcz_store.cc:276] 0504 20:00:11.456914 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.350336 (+893422us) service_pool.cc:224] Handling call
0504 20:00:12.407849 (+ 57513us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.414860 (+ 7011us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.576115 (+161255us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.639866 (+ 63751us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.640362 (+ 496us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":780032}
I20260504 20:00:12.695305 12888 tablet_service.cc:1511] Processing CreateTablet for tablet 431221deee84429b8261f87972c51218 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260504 20:00:12.667913 12740 rpcz_store.cc:276] 0504 20:00:11.392284 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.345027 (+952743us) service_pool.cc:224] Handling call
0504 20:00:12.346873 (+ 1846us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.346970 (+ 97us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.572551 (+225581us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.615516 (+ 42965us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.620261 (+ 4745us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.696064 12884 tablet_service.cc:1511] Processing CreateTablet for tablet 36b69b3d13384774a38b1442833e6948 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260504 20:00:12.666412 12964 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:12.641376 12880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2114545cfec4b19a5a3d1d6751f73da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.673842 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:12.696461 12716 leader_election.cc:343] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.697703 12825 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:12.697836 12883 tablet_service.cc:1511] Processing CreateTablet for tablet aada4371a0e5404bb6b69e5a2f3b1164 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260504 20:00:12.698974 12886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ea905fd808496c9b7d84a57c496cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.699120 12714 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:12.690424 12739 rpcz_store.cc:276] 0504 20:00:11.392893 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.416691 (+1023798us) service_pool.cc:224] Handling call
0504 20:00:12.418495 (+ 1804us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.418588 (+ 93us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.656193 (+237605us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.688898 (+ 32705us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.689470 (+ 572us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.700163 12950 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:12.700529 12950 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:12.700870 12950 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:12.701186 12739 tablet_service.cc:1511] Processing CreateTablet for tablet f2114545cfec4b19a5a3d1d6751f73da (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260504 20:00:12.702741 12740 tablet_service.cc:1511] Processing CreateTablet for tablet 91ff767ed7c24b97940a459421881055 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260504 20:00:12.703608 12888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431221deee84429b8261f87972c51218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.709115 12950 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.711385 12733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af25f180a18a4e79a93b35bb0a2e55f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.713400 12881 tablet_service.cc:1511] Processing CreateTablet for tablet 593a993b4ff341c9b98629e2c051ce00 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260504 20:00:12.667546 12732 rpcz_store.cc:276] 0504 20:00:11.392596 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.406367 (+1013771us) service_pool.cc:224] Handling call
0504 20:00:12.411303 (+ 4936us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.411405 (+ 102us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.622291 (+210886us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.664565 (+ 42274us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.665044 (+ 479us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.730366 12887 tablet_service.cc:1511] Processing CreateTablet for tablet 4318d305513d467c8254d7c054ef993e (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260504 20:00:12.731479 12740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ff767ed7c24b97940a459421881055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.733451 12885 tablet_service.cc:1511] Processing CreateTablet for tablet e67d58ddf5464b6daea5f094d46a05b6 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260504 20:00:12.733784 12880 tablet_service.cc:1511] Processing CreateTablet for tablet 640695edb69f427ea7660c9522bb01ef (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260504 20:00:12.734428 12737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43dc05259beb4569a29aeb5327c9b3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.678264 12731 rpcz_store.cc:276] 0504 20:00:11.393181 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.340719 (+947538us) service_pool.cc:224] Handling call
0504 20:00:12.344939 (+ 4220us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.345048 (+ 109us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.559934 (+214886us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.665390 (+105456us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.668315 (+ 2925us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.741240 12731 tablet_service.cc:1511] Processing CreateTablet for tablet 86ea905fd808496c9b7d84a57c496cb3 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260504 20:00:12.730962 12732 tablet_service.cc:1511] Processing CreateTablet for tablet b1271e7bfcea48b8bd072293da65c55d (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260504 20:00:12.742743 12731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ea905fd808496c9b7d84a57c496cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.720194 12940 tablet_bootstrap.cc:654] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.752545 12732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1271e7bfcea48b8bd072293da65c55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.761739 12944 tablet_bootstrap.cc:654] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.759459 12889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1271e7bfcea48b8bd072293da65c55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.770432 12739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2114545cfec4b19a5a3d1d6751f73da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.770712 12950 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:12.771592 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:12.771903 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:12.772239 12825 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:12.773473 12716 leader_election.cc:343] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.758056 12882 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37306 (request call id 44) took 1194 ms. Trace:
I20260504 20:00:12.778060 12882 rpcz_store.cc:276] 0504 20:00:11.562638 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.392270 (+829632us) service_pool.cc:224] Handling call
0504 20:00:12.446406 (+ 54136us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619f0ef1 00007f01619ce183 00007f016197cac0 00007f015adf2bf4 00007f015adf2b82 00007f015adf2ab1
0504 20:00:12.446548 (+ 142us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.446645 (+ 97us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.668991 (+222346us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.746274 (+ 77283us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.757339 (+ 11065us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10072320}
I20260504 20:00:12.780081 12881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593a993b4ff341c9b98629e2c051ce00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.788879 12887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4318d305513d467c8254d7c054ef993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.789546 12880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 640695edb69f427ea7660c9522bb01ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.794162 12884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b69b3d13384774a38b1442833e6948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.797055 12883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aada4371a0e5404bb6b69e5a2f3b1164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.798521 12885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67d58ddf5464b6daea5f094d46a05b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.811597 12939 tablet_bootstrap.cc:492] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:12.812186 12939 ts_tablet_manager.cc:1403] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.186s user 0.014s sys 0.000s
I20260504 20:00:12.819712 12940 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:12.822422 12735 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 44) took 1309 ms. Trace:
I20260504 20:00:12.822580 12940 ts_tablet_manager.cc:1403] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.168s user 0.013s sys 0.013s
I20260504 20:00:12.822927 12939 raft_consensus.cc:359] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.823668 12939 raft_consensus.cc:385] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:12.823972 12939 raft_consensus.cc:740] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.824743 12939 consensus_queue.cc:260] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.826964 12940 raft_consensus.cc:359] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.827872 12940 raft_consensus.cc:385] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:12.828125 12943 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:12.828361 12940 raft_consensus.cc:740] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.828637 12943 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.834203 12943 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:12.834890 12825 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:12.836098 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:12.836177 12714 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:12.837363 12950 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:12.837792 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:12.838042 12950 raft_consensus.cc:697] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:00:12.839504 12791 leader_election.cc:343] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:12.840212 12791 leader_election.cc:343] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.840657 12791 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:12.842032 12943 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:12.838977 12950 consensus_queue.cc:237] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.851117 12735 rpcz_store.cc:276] 0504 20:00:11.513233 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.492410 (+979177us) service_pool.cc:224] Handling call
0504 20:00:12.505406 (+ 12996us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.505523 (+ 117us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.751549 (+246026us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.813625 (+ 62076us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.822353 (+ 8728us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.872059 12939 ts_tablet_manager.cc:1434] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.060s user 0.005s sys 0.003s
I20260504 20:00:12.872694 12944 tablet_bootstrap.cc:492] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:12.872768 12734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 43) took 1452 ms. Trace:
I20260504 20:00:12.873256 12939 tablet_bootstrap.cc:492] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:12.873487 12944 ts_tablet_manager.cc:1403] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.285s user 0.028s sys 0.014s
I20260504 20:00:12.873991 12940 consensus_queue.cc:260] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.874879 12956 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:12.875353 12956 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.877010 12944 raft_consensus.cc:359] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.877746 12944 raft_consensus.cc:385] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:12.878118 12944 raft_consensus.cc:740] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.878849 12944 consensus_queue.cc:260] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.880402 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:12.881000 12944 ts_tablet_manager.cc:1434] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260504 20:00:12.883229 12956 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:12.885707 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:12.888224 12863 leader_election.cc:343] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.888680 12733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 45) took 1313 ms. Trace:
I20260504 20:00:12.873219 12734 rpcz_store.cc:276] 0504 20:00:11.420583 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.489590 (+1069007us) service_pool.cc:224] Handling call
0504 20:00:12.491557 (+ 1967us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.491651 (+ 94us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.771614 (+279963us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.872225 (+100611us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.872702 (+ 477us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.891598 12738 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 42) took 1498 ms. Trace:
I20260504 20:00:12.891990 12738 rpcz_store.cc:276] 0504 20:00:11.393468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.385221 (+991753us) service_pool.cc:224] Handling call
0504 20:00:12.490682 (+105461us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.490832 (+ 150us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.767075 (+276243us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.891103 (+124028us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.891538 (+ 435us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.889189 12733 rpcz_store.cc:276] 0504 20:00:11.575711 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.593905 (+1018194us) service_pool.cc:224] Handling call
0504 20:00:12.595949 (+ 2044us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.596146 (+ 197us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.777072 (+180926us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.888078 (+111006us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.888605 (+ 527us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.900478 12943 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:12.901012 12943 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.902441 12739 tablet_service.cc:1511] Processing CreateTablet for tablet 593a993b4ff341c9b98629e2c051ce00 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260504 20:00:12.903079 12943 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:12.903779 12739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593a993b4ff341c9b98629e2c051ce00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.904268 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:12.906062 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:12.907595 12791 leader_election.cc:343] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:12.908730 12791 leader_election.cc:343] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.909170 12791 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:12.910142 12944 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:12.911607 12940 ts_tablet_manager.cc:1434] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.089s user 0.007s sys 0.004s
I20260504 20:00:12.913131 12733 tablet_service.cc:1511] Processing CreateTablet for tablet aada4371a0e5404bb6b69e5a2f3b1164 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260504 20:00:12.914503 12733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aada4371a0e5404bb6b69e5a2f3b1164. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 20:00:12.914865 12865 leader_election.cc:343] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.915316 12865 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:12.916550 12943 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:12.917110 12944 tablet_bootstrap.cc:654] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.917471 12956 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:12.918632 12940 tablet_bootstrap.cc:492] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:12.919009 12950 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:12.919827 12731 tablet_service.cc:1511] Processing CreateTablet for tablet 4318d305513d467c8254d7c054ef993e (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260504 20:00:12.921186 12731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4318d305513d467c8254d7c054ef993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.920148 12950 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:12.924824 12740 tablet_service.cc:1511] Processing CreateTablet for tablet 431221deee84429b8261f87972c51218 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260504 20:00:12.933493 12740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431221deee84429b8261f87972c51218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.936383 12939 tablet_bootstrap.cc:654] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.937353 12734 tablet_service.cc:1511] Processing CreateTablet for tablet 36b69b3d13384774a38b1442833e6948 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260504 20:00:12.944959 12734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b69b3d13384774a38b1442833e6948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.940769 12736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 46) took 1125 ms. Trace:
I20260504 20:00:12.948508 12736 rpcz_store.cc:276] 0504 20:00:11.815341 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.604109 (+788768us) service_pool.cc:224] Handling call
0504 20:00:12.632159 (+ 28050us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.632986 (+ 827us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.853465 (+220479us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.940248 (+ 86783us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.940705 (+ 457us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.941978 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:12.950625 12714 leader_election.cc:343] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.941546 12737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 47) took 1014 ms. Trace:
I20260504 20:00:12.951272 12737 rpcz_store.cc:276] 0504 20:00:11.927240 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.679706 (+752466us) service_pool.cc:224] Handling call
0504 20:00:12.681618 (+ 1912us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.681728 (+ 110us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.874634 (+192906us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:12.913736 (+ 39102us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:12.941487 (+ 27751us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:12.969288 12940 tablet_bootstrap.cc:654] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:12.924813 12735 tablet_service.cc:1511] Processing CreateTablet for tablet 640695edb69f427ea7660c9522bb01ef (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260504 20:00:12.982103 12735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 640695edb69f427ea7660c9522bb01ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:12.983573 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:12.988646 12716 leader_election.cc:343] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:12.989351 12716 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:12.990675 12964 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:13.000315 12939 tablet_bootstrap.cc:492] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:13.000910 12939 ts_tablet_manager.cc:1403] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.128s user 0.018s sys 0.007s
I20260504 20:00:13.004534 12738 tablet_service.cc:1511] Processing CreateTablet for tablet e67d58ddf5464b6daea5f094d46a05b6 (DEFAULT_TABLE table=test-workload [id=8248412214354c89a76d2a364e6262e0]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260504 20:00:13.005656 12738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67d58ddf5464b6daea5f094d46a05b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:13.007490 12732 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:57450 (request call id 50) took 1060 ms. Trace:
I20260504 20:00:13.008040 12732 rpcz_store.cc:276] 0504 20:00:11.947346 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:12.730037 (+782691us) service_pool.cc:224] Handling call
0504 20:00:12.752037 (+ 22000us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 20:00:12.752165 (+ 128us) ts_tablet_manager.cc:682] Creating new metadata...
0504 20:00:12.839813 (+ 87648us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:13.006788 (+166975us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:13.007407 (+ 619us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:00:13.014426 12716 consensus_peers.cc:597] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Couldn't send request to peer 732c3560a688494196d22a1a48f89968. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:13.019579 12939 raft_consensus.cc:359] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:13.020498 12939 raft_consensus.cc:385] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.020790 12939 raft_consensus.cc:740] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:12.985409 12950 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.026142 12950 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:13.027853 12939 consensus_queue.cc:260] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:13.036278 12944 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:13.036870 12944 ts_tablet_manager.cc:1403] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.127s user 0.019s sys 0.013s
I20260504 20:00:13.044083 12956 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.044481 12944 raft_consensus.cc:359] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.044858 12956 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.045415 12939 ts_tablet_manager.cc:1434] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.043s user 0.006s sys 0.002s
I20260504 20:00:13.045601 12944 raft_consensus.cc:385] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.046075 12944 raft_consensus.cc:740] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.046541 12939 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:13.047195 12944 consensus_queue.cc:260] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.049239 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.050478 12944 ts_tablet_manager.cc:1434] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.013s user 0.006s sys 0.004s
I20260504 20:00:13.051914 12944 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:13.052083 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.047868 12956 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:13.054442 12863 leader_election.cc:343] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:13.055843 12865 leader_election.cc:343] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.056313 12865 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.056766 12939 tablet_bootstrap.cc:654] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.057624 12956 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:13.060166 12940 tablet_bootstrap.cc:492] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.062413 12944 tablet_bootstrap.cc:654] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.064059 12940 ts_tablet_manager.cc:1403] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.146s user 0.014s sys 0.018s
I20260504 20:00:13.068666 12940 raft_consensus.cc:359] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.085121 12940 raft_consensus.cc:385] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.085558 12940 raft_consensus.cc:740] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.086483 12940 consensus_queue.cc:260] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.092592 12940 ts_tablet_manager.cc:1434] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260504 20:00:13.100026 12944 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:13.100586 12944 ts_tablet_manager.cc:1403] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.049s user 0.032s sys 0.009s
I20260504 20:00:13.101302 12645 catalog_manager.cc:5671] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:13.105443 12939 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:13.105964 12939 ts_tablet_manager.cc:1403] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.017s
I20260504 20:00:13.108888 12940 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.108882 12939 raft_consensus.cc:359] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.109910 12939 raft_consensus.cc:385] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.110718 12939 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.111124 12944 raft_consensus.cc:359] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.111887 12944 raft_consensus.cc:385] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.111555 12939 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.112208 12944 raft_consensus.cc:740] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.113364 12944 consensus_queue.cc:260] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.117481 12944 ts_tablet_manager.cc:1434] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.017s user 0.004s sys 0.003s
I20260504 20:00:13.119007 12944 tablet_bootstrap.cc:492] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:13.119503 12940 tablet_bootstrap.cc:654] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.120764 12939 ts_tablet_manager.cc:1434] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260504 20:00:13.122385 12939 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:13.129912 12940 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.130815 12940 ts_tablet_manager.cc:1403] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.009s
I20260504 20:00:13.135821 12944 tablet_bootstrap.cc:654] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.136257 12940 raft_consensus.cc:359] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.137535 12940 raft_consensus.cc:385] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.138134 12940 raft_consensus.cc:740] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.139465 12940 consensus_queue.cc:260] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.146665 12940 ts_tablet_manager.cc:1434] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260504 20:00:13.148202 12939 tablet_bootstrap.cc:654] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.148270 12940 tablet_bootstrap.cc:492] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.163384 12940 tablet_bootstrap.cc:654] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.165273 12944 tablet_bootstrap.cc:492] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:13.166209 12944 ts_tablet_manager.cc:1403] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.029s
I20260504 20:00:13.169996 12939 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:13.170717 12939 ts_tablet_manager.cc:1403] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.015s
I20260504 20:00:13.171317 12944 raft_consensus.cc:359] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.172273 12944 raft_consensus.cc:385] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.172609 12944 raft_consensus.cc:740] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.173404 12944 consensus_queue.cc:260] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.176121 12944 ts_tablet_manager.cc:1434] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260504 20:00:13.175751 12939 raft_consensus.cc:359] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.176767 12939 raft_consensus.cc:385] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.177211 12939 raft_consensus.cc:740] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.177875 12944 tablet_bootstrap.cc:492] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:13.178045 12940 tablet_bootstrap.cc:492] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.178798 12940 ts_tablet_manager.cc:1403] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.013s
I20260504 20:00:13.178470 12939 consensus_queue.cc:260] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.184514 12940 raft_consensus.cc:359] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.185370 12943 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.186069 12940 raft_consensus.cc:385] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.186211 12943 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.186602 12940 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.188055 12940 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.190301 12943 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.190364 12944 tablet_bootstrap.cc:654] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.190593 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.191663 12943 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.192448 12943 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:13.194608 12939 ts_tablet_manager.cc:1434] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.023s user 0.009s sys 0.003s
I20260504 20:00:13.191689 12940 ts_tablet_manager.cc:1434] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260504 20:00:13.196550 12943 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:13.196696 12939 tablet_bootstrap.cc:492] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:13.197429 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:13.191596 12750 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:13.198438 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.199272 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:13.200318 12898 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:13.200246 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:13.201821 12899 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
W20260504 20:00:13.202194 12791 leader_election.cc:343] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.203092 12943 raft_consensus.cc:2804] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.203620 12943 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.204116 12943 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.207291 12791 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.218750 12940 tablet_bootstrap.cc:492] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.224097 12956 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.224897 12956 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.222426 12943 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.233309 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.233867 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.234225 12749 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:13.231073 12939 tablet_bootstrap.cc:654] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.236111 12944 tablet_bootstrap.cc:492] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
W20260504 20:00:13.236390 12863 leader_election.cc:343] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.236986 12944 ts_tablet_manager.cc:1403] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.059s user 0.027s sys 0.023s
I20260504 20:00:13.238420 12865 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:13.239475 12940 tablet_bootstrap.cc:654] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.228896 12956 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.246662 12944 raft_consensus.cc:359] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.248040 12944 raft_consensus.cc:385] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.248559 12944 raft_consensus.cc:740] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.248737 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:13.242800 12956 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.249964 12749 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.250025 12956 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.251809 12956 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.259326 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:13.260396 12899 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.260706 12939 tablet_bootstrap.cc:492] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:13.261428 12939 ts_tablet_manager.cc:1403] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.065s user 0.016s sys 0.008s
I20260504 20:00:13.251201 12943 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.250685 12944 consensus_queue.cc:260] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.264534 12987 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.268213 12944 ts_tablet_manager.cc:1434] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.031s user 0.007s sys 0.007s
I20260504 20:00:13.270679 12956 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.272342 12939 raft_consensus.cc:359] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.274278 12939 raft_consensus.cc:385] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.274561 12939 raft_consensus.cc:740] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.275453 12899 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:13.277386 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:13.278098 12940 tablet_bootstrap.cc:492] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.278697 12940 ts_tablet_manager.cc:1403] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.026s
I20260504 20:00:13.278656 12943 raft_consensus.cc:2804] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.279103 12939 consensus_queue.cc:260] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.281455 12944 tablet_bootstrap.cc:492] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:13.284422 12940 raft_consensus.cc:359] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.285372 12940 raft_consensus.cc:385] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.285758 12940 raft_consensus.cc:740] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.286824 12940 consensus_queue.cc:260] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.274060 12987 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.290377 12940 ts_tablet_manager.cc:1434] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260504 20:00:13.290356 12987 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.290825 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:13.291651 12750 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.292160 12940 tablet_bootstrap.cc:492] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.302718 12899 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.304936 12940 tablet_bootstrap.cc:654] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.311457 12956 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.322333 12749 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:13.323666 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:13.324802 12987 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:13.326169 12956 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.326970 12956 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:13.330673 12863 leader_election.cc:343] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.332089 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.334205 12943 raft_consensus.cc:697] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:13.336846 12749 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:13.339228 12865 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:13.341138 12988 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.341761 12988 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.340034 12943 consensus_queue.cc:237] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.342473 12988 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.342653 12750 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:13.335696 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.333508 12956 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:13.347720 12863 leader_election.cc:343] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.350190 12865 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:13.351487 12956 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.351444 12987 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:13.355202 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:13.356694 12991 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.357710 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:13.357560 12991 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.358765 12898 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.361182 12991 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:13.363480 12791 leader_election.cc:343] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.365765 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.366916 12750 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:13.368734 12791 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:13.374583 12991 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.376052 12939 ts_tablet_manager.cc:1434] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.114s user 0.019s sys 0.049s
I20260504 20:00:13.375857 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:13.377195 12939 tablet_bootstrap.cc:492] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:13.377290 12956 raft_consensus.cc:697] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:00:13.378400 12791 leader_election.cc:343] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.378546 12988 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.376111 12991 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.383376 12991 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.384689 12940 tablet_bootstrap.cc:492] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.385516 12940 ts_tablet_manager.cc:1403] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.094s user 0.015s sys 0.012s
I20260504 20:00:13.386852 12898 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:13.388757 12939 tablet_bootstrap.cc:654] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.388548 12791 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.393734 12940 raft_consensus.cc:359] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.395114 12940 raft_consensus.cc:385] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.395571 12940 raft_consensus.cc:740] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.399269 12991 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.400046 12993 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.401456 12944 tablet_bootstrap.cc:654] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.401333 12993 raft_consensus.cc:697] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:13.396509 12940 consensus_queue.cc:260] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.378880 12956 consensus_queue.cc:237] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.407352 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:13.408324 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:13.408200 12964 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:13.408383 12750 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.409233 12749 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.404359 12988 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.414731 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:13.415530 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:00:13.416625 12791 leader_election.cc:343] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:13.419016 12863 leader_election.cc:343] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.412770 12991 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.437465 12964 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:13.416302 12993 consensus_queue.cc:237] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:13.447657 12964 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.450881 12940 ts_tablet_manager.cc:1434] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.061s user 0.017s sys 0.013s
I20260504 20:00:13.454952 12964 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.447834 12749 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:13.456975 12940 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.457046 12939 tablet_bootstrap.cc:492] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:13.458849 12939 ts_tablet_manager.cc:1403] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.013s
I20260504 20:00:13.457674 12646 catalog_manager.cc:5671] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:13.462131 12791 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:13.463676 12943 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.464504 12943 raft_consensus.cc:697] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:13.466151 12943 consensus_queue.cc:237] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.466776 12939 raft_consensus.cc:359] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.467474 12939 raft_consensus.cc:385] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.466962 12648 catalog_manager.cc:5671] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:13.467738 12939 raft_consensus.cc:740] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.468500 12939 consensus_queue.cc:260] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.472882 12939 ts_tablet_manager.cc:1434] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260504 20:00:13.474164 12939 tablet_bootstrap.cc:492] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:13.475051 12964 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.476315 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.476998 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:13.477142 12825 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:13.478341 12940 tablet_bootstrap.cc:654] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:13.478914 12716 leader_election.cc:343] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.488897 12714 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:13.490134 12964 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.490693 12964 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.491252 12964 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.492863 12750 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:13.524080 12865 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:13.526044 12998 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.526813 12998 raft_consensus.cc:697] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:13.500878 12944 tablet_bootstrap.cc:492] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:13.528291 12998 consensus_queue.cc:237] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.509555 12964 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:13.529299 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:13.500975 12939 tablet_bootstrap.cc:654] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.532522 12964 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.529295 12944 ts_tablet_manager.cc:1403] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.248s user 0.015s sys 0.011s
I20260504 20:00:13.537029 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:00:13.540076 12716 leader_election.cc:343] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.541551 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:13.542613 12825 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.551769 12940 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.552407 12940 ts_tablet_manager.cc:1403] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.096s user 0.024s sys 0.021s
I20260504 20:00:13.556145 12825 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:13.558789 12944 raft_consensus.cc:359] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:13.558350 12714 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:13.559854 12944 raft_consensus.cc:385] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.560211 12944 raft_consensus.cc:740] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.560250 12964 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.561298 12964 raft_consensus.cc:697] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:13.561111 12944 consensus_queue.cc:260] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:13.562515 12964 consensus_queue.cc:237] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.563879 12944 ts_tablet_manager.cc:1434] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.028s user 0.009s sys 0.001s
I20260504 20:00:13.555670 12940 raft_consensus.cc:359] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.565635 12940 raft_consensus.cc:385] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.565964 12943 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.565995 12940 raft_consensus.cc:740] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.566642 12943 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.567008 12940 consensus_queue.cc:260] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.569576 12940 ts_tablet_manager.cc:1434] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20260504 20:00:13.569429 12943 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.570968 12940 tablet_bootstrap.cc:492] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.555644 12647 catalog_manager.cc:5671] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:13.575172 12944 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:13.575234 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:13.579532 12791 leader_election.cc:343] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.583478 12646 catalog_manager.cc:5671] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:13.587900 12940 tablet_bootstrap.cc:654] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.588732 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:13.589610 12899 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:13.587908 12944 tablet_bootstrap.cc:654] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.590080 12939 tablet_bootstrap.cc:492] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:13.591333 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.591405 12898 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.593348 12943 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.593993 12943 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.594563 12943 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.600663 12964 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.610285 12964 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:13.614771 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:13.616214 12899 raft_consensus.cc:2393] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:13.620187 12943 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.627357 12939 ts_tablet_manager.cc:1403] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.153s user 0.030s sys 0.028s
I20260504 20:00:13.628206 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.629364 12647 catalog_manager.cc:5671] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:13.630802 12714 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:13.632061 12950 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.635293 12943 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.638185 12939 raft_consensus.cc:359] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.639006 12939 raft_consensus.cc:385] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.639338 12939 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.640460 12939 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.645076 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:13.645618 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:13.646510 12899 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:13.647505 12791 leader_election.cc:343] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.651294 13000 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.652058 13000 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.641615 12964 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.651580 12939 ts_tablet_manager.cc:1434] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20260504 20:00:13.656605 13000 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.656733 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.658288 12939 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
W20260504 20:00:13.661495 12865 leader_election.cc:343] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.662876 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.665581 12998 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:13.666373 12998 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.667579 12950 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:13.671475 12940 tablet_bootstrap.cc:492] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.672588 12940 ts_tablet_manager.cc:1403] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.102s user 0.014s sys 0.018s
I20260504 20:00:13.672092 12998 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.673736 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.675642 12940 raft_consensus.cc:359] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.676481 12940 raft_consensus.cc:385] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.676882 12940 raft_consensus.cc:740] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.677651 12940 consensus_queue.cc:260] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.676584 12939 tablet_bootstrap.cc:654] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.678431 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.665190 12824 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:13.682175 12863 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.682368 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.684729 13000 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.685207 13000 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.685642 13000 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.685860 12940 ts_tablet_manager.cc:1434] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260504 20:00:13.686998 12940 tablet_bootstrap.cc:492] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.687052 12899 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:13.691298 12995 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:13.696743 12940 tablet_bootstrap.cc:654] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.700274 12944 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:13.702786 13000 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.708241 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:13.709092 12944 ts_tablet_manager.cc:1403] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.134s user 0.025s sys 0.015s
W20260504 20:00:13.710404 12865 leader_election.cc:343] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.711843 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.712894 12943 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.713572 12943 raft_consensus.cc:697] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:13.714241 13000 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.714958 12943 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.719172 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:13.719103 12944 raft_consensus.cc:359] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.719929 12944 raft_consensus.cc:385] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.719964 12824 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.720340 12944 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.721097 12944 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.727552 12944 ts_tablet_manager.cc:1434] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260504 20:00:13.735136 12944 tablet_bootstrap.cc:492] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:13.737435 12824 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:13.746591 12863 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.748888 13000 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.749683 13000 raft_consensus.cc:697] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:13.748245 12644 catalog_manager.cc:5671] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:13.752202 13000 consensus_queue.cc:237] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.759699 12943 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.760867 12939 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:13.761459 12939 ts_tablet_manager.cc:1403] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.103s user 0.039s sys 0.039s
I20260504 20:00:13.760335 12943 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.765357 12940 tablet_bootstrap.cc:492] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.766350 12939 raft_consensus.cc:359] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.766909 12939 raft_consensus.cc:385] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.767166 12939 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.770565 12645 catalog_manager.cc:5671] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:13.773659 12939 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.775203 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.773721 12940 ts_tablet_manager.cc:1403] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.087s user 0.028s sys 0.003s
I20260504 20:00:13.776017 12939 ts_tablet_manager.cc:1434] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260504 20:00:13.777033 12939 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:13.777429 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:13.777928 12791 leader_election.cc:343] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:13.779093 12791 leader_election.cc:343] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.783560 12791 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.784024 12940 raft_consensus.cc:359] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.784901 12940 raft_consensus.cc:385] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.784806 12993 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:13.776543 12943 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.785384 12940 raft_consensus.cc:740] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.786226 12940 consensus_queue.cc:260] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.788859 12940 ts_tablet_manager.cc:1434] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260504 20:00:13.791816 12944 tablet_bootstrap.cc:654] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.793290 13000 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.794035 13000 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.795230 12939 tablet_bootstrap.cc:654] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.798820 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:13.801118 12865 leader_election.cc:343] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.804176 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.805119 12824 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:13.807403 12863 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.808913 13000 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.808955 12998 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:13.812074 12998 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:13.812589 12998 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.818113 12993 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:13.824752 12940 tablet_bootstrap.cc:492] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.835250 12645 catalog_manager.cc:5671] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:13.840116 13000 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.840679 12940 tablet_bootstrap.cc:654] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:13.840945 13000 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.848062 13000 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.849259 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.848973 12998 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.850538 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:13.851716 12824 raft_consensus.cc:2393] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 1: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:00:13.852368 12998 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:13.853371 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:13.854271 12863 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.856021 12998 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:13.855512 12865 leader_election.cc:343] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:13.857913 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:13.858738 12824 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.862864 12993 consensus_queue.cc:1048] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:13.871263 12993 consensus_queue.cc:1048] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260504 20:00:13.876950 12998 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:13.873871 12939 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:13.878327 12939 ts_tablet_manager.cc:1403] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.102s user 0.029s sys 0.017s
I20260504 20:00:13.885497 12939 raft_consensus.cc:359] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.886199 12939 raft_consensus.cc:385] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.886523 12939 raft_consensus.cc:740] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.887317 12939 consensus_queue.cc:260] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.903945 12939 ts_tablet_manager.cc:1434] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.025s user 0.008s sys 0.002s
I20260504 20:00:13.903375 12824 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:13.905700 12993 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.906340 12993 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.907286 12939 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:13.909363 12993 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:13.911127 12863 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.915335 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:13.913676 12998 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:13.919478 12998 raft_consensus.cc:697] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:13.926887 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:13.920818 12998 consensus_queue.cc:237] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.928354 12750 raft_consensus.cc:2393] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:13.930233 12992 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:13.933166 12899 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.937368 12791 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:13.940438 12992 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:13.941702 12940 tablet_bootstrap.cc:492] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:13.942931 12940 ts_tablet_manager.cc:1403] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.118s user 0.021s sys 0.019s
I20260504 20:00:13.947037 12940 raft_consensus.cc:359] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.947890 12940 raft_consensus.cc:385] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:13.948330 12940 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:13.949379 12940 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:13.951733 12940 ts_tablet_manager.cc:1434] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260504 20:00:13.979727 12940 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:13.995904 12944 tablet_bootstrap.cc:492] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:13.997062 12944 ts_tablet_manager.cc:1403] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.262s user 0.042s sys 0.062s
I20260504 20:00:13.995872 13000 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:13.998396 13000 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.004477 12993 consensus_queue.cc:1048] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.009436 12939 tablet_bootstrap.cc:654] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.011229 12995 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.011761 12992 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.013123 12944 raft_consensus.cc:359] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.019944 12944 raft_consensus.cc:385] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.020303 12944 raft_consensus.cc:740] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.020916 12944 consensus_queue.cc:260] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.022547 12944 ts_tablet_manager.cc:1434] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260504 20:00:14.023224 12944 tablet_bootstrap.cc:492] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:14.026322 12998 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.027237 13000 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:14.018643 12964 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.033720 12940 tablet_bootstrap.cc:654] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.035205 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:14.030576 12964 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.054059 12964 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.058548 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:14.058888 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:14.061014 12944 tablet_bootstrap.cc:654] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.086097 12863 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.086501 12992 consensus_queue.cc:1048] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.089041 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.090051 12898 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:14.087991 13000 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:14.091070 12998 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.100564 12998 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.104126 12998 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:14.112802 12716 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:14.113449 12940 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:14.113919 12940 ts_tablet_manager.cc:1403] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.135s user 0.017s sys 0.018s
I20260504 20:00:14.114073 12964 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.137838 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:14.139086 12825 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:14.142933 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:14.149518 12865 leader_election.cc:343] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.151473 12940 raft_consensus.cc:359] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.152565 12940 raft_consensus.cc:385] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.152930 12940 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.157150 12940 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.173266 12940 ts_tablet_manager.cc:1434] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.059s user 0.009s sys 0.000s
I20260504 20:00:14.174337 12940 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:14.182804 12964 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.183369 12939 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:14.186695 12939 ts_tablet_manager.cc:1403] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.280s user 0.023s sys 0.055s
I20260504 20:00:14.167409 12863 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.184291 13000 consensus_queue.cc:1048] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.189561 12644 catalog_manager.cc:5671] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:14.191130 12939 raft_consensus.cc:359] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.191922 12939 raft_consensus.cc:385] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.192234 12939 raft_consensus.cc:740] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.192962 12939 consensus_queue.cc:260] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.200634 12995 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:14.201213 12995 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:00:14.201674 12863 consensus_peers.cc:597] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 -> Peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Couldn't send request to peer 5ff0ef72ec3b4667893a0420d3c7f923. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:14.201685 12995 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.211206 12940 tablet_bootstrap.cc:654] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.199769 12992 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260504 20:00:14.202376 12964 consensus_queue.cc:1048] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260504 20:00:14.226459 13000 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.227761 12939 ts_tablet_manager.cc:1434] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.041s user 0.004s sys 0.004s
I20260504 20:00:14.228775 12939 tablet_bootstrap.cc:492] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:14.227756 13033 consensus_queue.cc:1048] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260504 20:00:14.241426 12992 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.256412 12995 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.260784 12944 tablet_bootstrap.cc:492] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:14.267982 12992 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.275133 12992 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.305371 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:14.309404 12791 leader_election.cc:343] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.312371 12992 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.321514 12995 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:14.324198 12944 ts_tablet_manager.cc:1403] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.301s user 0.032s sys 0.015s
I20260504 20:00:14.326695 12939 tablet_bootstrap.cc:654] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.333580 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:14.334563 12823 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.335244 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.336133 12897 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:14.340068 12791 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.341106 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:14.342926 13025 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:14.343528 13025 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:14.344096 13025 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.348896 12940 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:14.349500 12940 ts_tablet_manager.cc:1403] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.175s user 0.005s sys 0.014s
W20260504 20:00:14.351836 12865 consensus_peers.cc:597] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 -> Peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Couldn't send request to peer 82787d8e64d24ca885ca517aa8ed1be2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 20:00:14.352807 12865 leader_election.cc:343] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.354172 12964 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.350970 12944 raft_consensus.cc:359] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.363374 12944 raft_consensus.cc:385] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.363775 12944 raft_consensus.cc:740] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.364442 12964 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.364492 12944 consensus_queue.cc:260] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.369177 12964 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.371799 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:14.374459 12714 leader_election.cc:343] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.378950 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.396389 13025 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.401201 13041 consensus_queue.cc:1048] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260504 20:00:14.403703 12716 leader_election.cc:343] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.404508 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:14.405663 12950 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.399854 12940 raft_consensus.cc:359] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.407390 12940 raft_consensus.cc:385] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.407701 12940 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.409348 12940 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.411916 12944 ts_tablet_manager.cc:1434] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.087s user 0.008s sys 0.003s
I20260504 20:00:14.412708 12944 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:14.424595 12940 ts_tablet_manager.cc:1434] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.075s user 0.009s sys 0.000s
I20260504 20:00:14.425694 12940 tablet_bootstrap.cc:492] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:14.434763 13025 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.438304 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:14.438717 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:14.439576 12897 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.437528 12939 tablet_bootstrap.cc:492] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
W20260504 20:00:14.452211 12791 leader_election.cc:343] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.451864 12944 tablet_bootstrap.cc:654] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.454521 12939 ts_tablet_manager.cc:1403] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.226s user 0.016s sys 0.016s
I20260504 20:00:14.458395 12950 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.459268 12950 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.463910 12950 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.458643 12823 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:14.464232 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:14.465947 12863 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.467806 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.468252 13041 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:14.469007 13041 raft_consensus.cc:697] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:14.470515 12716 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:14.472204 12950 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.470503 13041 consensus_queue.cc:237] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.479765 12897 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:14.486462 12791 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.480109 12939 raft_consensus.cc:359] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.489289 12939 raft_consensus.cc:385] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.489646 12939 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.489754 13045 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:14.490453 12939 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.490641 13045 raft_consensus.cc:697] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:14.497720 12749 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.503086 13000 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.503984 13000 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.492053 13045 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.509765 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:14.511960 12865 leader_election.cc:343] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.515280 13000 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:14.520799 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:14.523572 12940 tablet_bootstrap.cc:654] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.526788 12939 ts_tablet_manager.cc:1434] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.071s user 0.015s sys 0.001s
I20260504 20:00:14.529742 12939 tablet_bootstrap.cc:492] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
W20260504 20:00:14.532785 12863 leader_election.cc:343] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.533270 12863 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.534566 13041 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.535737 12944 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:14.536269 12944 ts_tablet_manager.cc:1403] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.124s user 0.021s sys 0.019s
I20260504 20:00:14.538720 13041 consensus_queue.cc:1048] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.539800 12944 raft_consensus.cc:359] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.540817 12944 raft_consensus.cc:385] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.541152 12944 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.541937 12944 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.544549 12944 ts_tablet_manager.cc:1434] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260504 20:00:14.545622 12944 tablet_bootstrap.cc:492] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:14.549527 12939 tablet_bootstrap.cc:654] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:14.567921 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:14.571204 12940 tablet_bootstrap.cc:492] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:14.571803 12940 ts_tablet_manager.cc:1403] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.146s user 0.021s sys 0.016s
I20260504 20:00:14.580334 12944 tablet_bootstrap.cc:654] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.580204 12940 raft_consensus.cc:359] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.580988 12940 raft_consensus.cc:385] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.581323 12940 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.582386 12940 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.583957 12940 ts_tablet_manager.cc:1434] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260504 20:00:14.584746 12940 tablet_bootstrap.cc:492] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:14.593070 12939 tablet_bootstrap.cc:492] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:14.593734 12939 ts_tablet_manager.cc:1403] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.064s user 0.015s sys 0.002s
I20260504 20:00:14.594813 12940 tablet_bootstrap.cc:654] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.597204 12939 raft_consensus.cc:359] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.598241 12939 raft_consensus.cc:385] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.598616 12939 raft_consensus.cc:740] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.599990 12939 consensus_queue.cc:260] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.604493 12646 catalog_manager.cc:5671] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:14.615944 13045 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.616652 13045 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.619972 13045 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.621640 13041 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.622090 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.622359 13041 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.623521 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:14.624552 12897 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:14.624809 13045 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:00:14.625473 12791 leader_election.cc:343] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.627491 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:14.628336 12825 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:14.625473 13045 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.629899 12863 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:14.631080 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:14.631155 13000 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:14.631331 12791 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.631963 13000 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:14.632484 13000 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.632671 12993 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:14.633028 13041 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:14.633550 12865 leader_election.cc:343] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.634735 12993 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:14.635298 12993 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.639767 12944 tablet_bootstrap.cc:492] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:14.640687 12944 ts_tablet_manager.cc:1403] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.095s user 0.030s sys 0.020s
I20260504 20:00:14.641753 13000 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.643738 12648 catalog_manager.cc:5671] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:14.644831 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:14.645174 12944 raft_consensus.cc:359] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.645867 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:14.646150 12944 raft_consensus.cc:385] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.646471 12944 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
W20260504 20:00:14.646399 12791 leader_election.cc:343] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.647395 12944 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.649019 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:14.650127 12944 ts_tablet_manager.cc:1434] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260504 20:00:14.650161 12750 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.651361 12944 tablet_bootstrap.cc:492] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:14.654402 13045 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.655609 12939 ts_tablet_manager.cc:1434] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.061s user 0.008s sys 0.000s
I20260504 20:00:14.662194 12939 tablet_bootstrap.cc:492] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:14.665758 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:14.668089 12791 leader_election.cc:343] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.668855 12791 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.674841 13045 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.660282 13000 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:14.676283 12993 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.678188 12750 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:14.679819 12825 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:14.683866 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:14.684533 12944 tablet_bootstrap.cc:654] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.684857 12939 tablet_bootstrap.cc:654] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.689056 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:14.690276 12897 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:14.692546 12791 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.693902 13045 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.694290 12992 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.691008 12993 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.694916 12992 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.697322 12940 tablet_bootstrap.cc:492] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:14.697744 12940 ts_tablet_manager.cc:1403] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.113s user 0.015s sys 0.003s
I20260504 20:00:14.702227 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:14.704064 13000 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:14.704599 12992 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.704794 13000 raft_consensus.cc:697] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:14.705737 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:14.706161 13000 consensus_queue.cc:237] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.702240 12944 tablet_bootstrap.cc:492] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:14.708348 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.713435 12944 ts_tablet_manager.cc:1403] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.017s
I20260504 20:00:14.713711 12791 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:14.716195 12992 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.718855 12939 tablet_bootstrap.cc:492] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:14.719478 12939 ts_tablet_manager.cc:1403] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.059s user 0.017s sys 0.007s
I20260504 20:00:14.719684 12944 raft_consensus.cc:359] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.720808 12944 raft_consensus.cc:385] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.721040 12825 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.721194 12944 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.722196 12944 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.723567 12939 raft_consensus.cc:359] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.724412 12939 raft_consensus.cc:385] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.724695 12939 raft_consensus.cc:740] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.725548 12939 consensus_queue.cc:260] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.733922 12940 raft_consensus.cc:359] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.739596 12940 raft_consensus.cc:385] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.734917 12944 ts_tablet_manager.cc:1434] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.021s user 0.009s sys 0.000s
I20260504 20:00:14.739099 12645 catalog_manager.cc:5671] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:14.741549 12944 tablet_bootstrap.cc:492] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:14.740242 12940 raft_consensus.cc:740] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.745465 12939 ts_tablet_manager.cc:1434] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260504 20:00:14.746577 12939 tablet_bootstrap.cc:492] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:14.749938 12940 consensus_queue.cc:260] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.752547 12940 ts_tablet_manager.cc:1434] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20260504 20:00:14.753455 12944 tablet_bootstrap.cc:654] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.753743 12940 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:14.765350 12940 tablet_bootstrap.cc:654] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.780678 12944 tablet_bootstrap.cc:492] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:14.782298 12944 ts_tablet_manager.cc:1403] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.011s
I20260504 20:00:14.781078 12939 tablet_bootstrap.cc:654] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.788331 12940 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:14.789041 12940 ts_tablet_manager.cc:1403] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.009s
I20260504 20:00:14.792183 12944 raft_consensus.cc:359] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.792971 12940 raft_consensus.cc:359] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.793692 12944 raft_consensus.cc:385] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.793829 12940 raft_consensus.cc:385] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.794162 12944 raft_consensus.cc:740] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.794544 12940 raft_consensus.cc:740] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.795045 12944 consensus_queue.cc:260] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.795387 12940 consensus_queue.cc:260] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.798148 12944 ts_tablet_manager.cc:1434] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.015s user 0.009s sys 0.002s
I20260504 20:00:14.799784 12944 tablet_bootstrap.cc:492] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:14.803133 12940 ts_tablet_manager.cc:1434] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.014s user 0.005s sys 0.006s
I20260504 20:00:14.804418 12940 tablet_bootstrap.cc:492] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:14.807474 12950 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.808163 12950 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.809444 12944 tablet_bootstrap.cc:654] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.812209 12950 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.816469 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:14.818850 12716 leader_election.cc:343] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.814586 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:14.820099 12940 tablet_bootstrap.cc:654] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:14.822806 12714 leader_election.cc:343] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.823820 12714 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:14.824988 12992 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.825904 12992 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.829190 12939 tablet_bootstrap.cc:492] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:14.829864 12939 ts_tablet_manager.cc:1403] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.084s user 0.019s sys 0.011s
I20260504 20:00:14.830567 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:14.831637 12750 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:14.833583 12992 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.833851 12939 raft_consensus.cc:359] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.835713 12939 raft_consensus.cc:385] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.835989 12939 raft_consensus.cc:740] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.835289 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.836905 12950 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:14.837090 12791 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:14.837512 12897 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:14.837786 12950 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.836433 12939 consensus_queue.cc:260] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.839121 12993 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:14.838729 12950 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.841377 12939 ts_tablet_manager.cc:1434] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260504 20:00:14.842597 12939 tablet_bootstrap.cc:492] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:14.848595 12993 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:14.850267 12993 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.854002 12944 tablet_bootstrap.cc:492] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:14.854868 12944 ts_tablet_manager.cc:1403] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.055s user 0.014s sys 0.016s
I20260504 20:00:14.856559 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:14.858340 12714 leader_election.cc:343] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.860795 12939 tablet_bootstrap.cc:654] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.866531 12950 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.866982 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.869946 12897 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:14.870545 12944 raft_consensus.cc:359] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.871308 12944 raft_consensus.cc:385] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.871798 12944 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.871433 12716 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:14.872589 12944 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.873912 12950 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:14.874696 12950 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:14.875172 12944 ts_tablet_manager.cc:1434] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260504 20:00:14.875409 12940 tablet_bootstrap.cc:492] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:14.875926 12940 ts_tablet_manager.cc:1403] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.008s
I20260504 20:00:14.876194 12944 tablet_bootstrap.cc:492] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:14.879482 12940 raft_consensus.cc:359] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.880555 12940 raft_consensus.cc:385] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.881003 12940 raft_consensus.cc:740] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.875234 12950 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.882184 12940 consensus_queue.cc:260] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.888453 12944 tablet_bootstrap.cc:654] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.894995 12993 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.897517 12940 ts_tablet_manager.cc:1434] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.021s user 0.009s sys 0.002s
I20260504 20:00:14.898936 12940 tablet_bootstrap.cc:492] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:14.916581 12939 tablet_bootstrap.cc:492] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:14.917274 12939 ts_tablet_manager.cc:1403] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.075s user 0.021s sys 0.012s
I20260504 20:00:14.922470 12940 tablet_bootstrap.cc:654] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:14.925379 12993 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.926421 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:14.927847 12749 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.929044 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:14.930050 12897 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.926030 12950 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.937357 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:14.937657 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:14.938444 12899 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:14.939512 12714 leader_election.cc:343] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:14.941028 12950 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.941092 12939 raft_consensus.cc:359] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.943933 12939 raft_consensus.cc:385] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.945956 12939 raft_consensus.cc:740] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.947165 12939 consensus_queue.cc:260] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.951671 12939 ts_tablet_manager.cc:1434] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.034s user 0.010s sys 0.001s
I20260504 20:00:14.953405 12939 tablet_bootstrap.cc:492] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:14.949005 12749 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:14.955166 12791 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:14.956281 13045 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:14.956897 13045 raft_consensus.cc:697] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:14.958086 13045 consensus_queue.cc:237] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.960659 12944 tablet_bootstrap.cc:492] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:14.953181 12897 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:14.949074 12899 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:14.965330 12716 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:14.967938 12950 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:14.969058 12944 ts_tablet_manager.cc:1403] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.093s user 0.023s sys 0.022s
I20260504 20:00:14.968987 12950 raft_consensus.cc:697] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:14.970747 12964 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.971985 12964 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.974372 12944 raft_consensus.cc:359] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.975169 12944 raft_consensus.cc:385] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:14.975478 12944 raft_consensus.cc:740] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:14.975826 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:14.976274 12944 consensus_queue.cc:260] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.976974 12825 raft_consensus.cc:2393] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:14.979640 12964 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:14.981796 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:14.982446 12950 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.982805 13000 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:14.983405 12944 ts_tablet_manager.cc:1434] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260504 20:00:14.984490 12944 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:14.984452 12716 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:14.988662 12964 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:14.992129 12993 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:14.992937 12993 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:14.995800 12939 tablet_bootstrap.cc:654] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.004227 12940 tablet_bootstrap.cc:492] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:15.004844 12940 ts_tablet_manager.cc:1403] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.106s user 0.017s sys 0.030s
I20260504 20:00:15.006137 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:15.010305 12749 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:15.014496 12791 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:15.015731 12992 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:15.016243 12992 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:15.016717 12992 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.017024 12940 raft_consensus.cc:359] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.017912 12940 raft_consensus.cc:385] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.018245 12940 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.018481 12646 catalog_manager.cc:5671] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:15.018977 12940 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.029321 12940 ts_tablet_manager.cc:1434] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20260504 20:00:15.030788 12940 tablet_bootstrap.cc:492] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:15.031172 12950 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.033622 12944 tablet_bootstrap.cc:654] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.033634 12950 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.038442 12992 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.041498 12992 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.042359 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:15.042632 12950 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.043697 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:15.046247 12716 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:15.048273 12950 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:15.052457 12964 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:15.056306 12940 tablet_bootstrap.cc:654] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.057549 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:15.058353 12748 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.060760 13000 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:15.063438 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:15.062695 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:15.065522 12939 tablet_bootstrap.cc:492] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:15.066057 12939 ts_tablet_manager.cc:1403] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.113s user 0.018s sys 0.012s
I20260504 20:00:15.070112 12939 raft_consensus.cc:359] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.070879 12939 raft_consensus.cc:385] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.071204 12939 raft_consensus.cc:740] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.072037 12939 consensus_queue.cc:260] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.074510 12964 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.074875 12939 ts_tablet_manager.cc:1434] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260504 20:00:15.075316 12964 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.076000 12939 tablet_bootstrap.cc:492] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
W20260504 20:00:15.071341 12791 leader_election.cc:343] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.078251 12993 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:15.078912 12791 leader_election.cc:343] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.079397 13000 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.079955 13000 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.080670 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:15.083041 12714 leader_election.cc:343] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.084422 12964 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.085798 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:15.086778 12898 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:15.086009 12645 catalog_manager.cc:5671] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:15.091253 12748 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:15.096975 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:15.098794 12791 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
W20260504 20:00:15.099493 12865 leader_election.cc:343] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.099773 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:15.101437 12993 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:15.103399 12993 raft_consensus.cc:697] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:15.103601 13000 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:15.104580 12993 consensus_queue.cc:237] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.105268 12716 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:15.106130 12964 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:15.106053 12823 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:15.106686 12964 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:15.107172 12964 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.108011 12863 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:15.117997 12964 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.123008 12992 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:15.126940 12964 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.127211 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:15.128369 12897 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.129345 12939 tablet_bootstrap.cc:654] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.133925 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:00:15.135553 12714 leader_election.cc:343] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:15.136188 12791 consensus_peers.cc:597] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Couldn't send request to peer 82787d8e64d24ca885ca517aa8ed1be2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:15.154408 13041 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:15.154978 13041 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:15.155398 13041 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.171180 12944 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:15.171862 12944 ts_tablet_manager.cc:1403] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.188s user 0.020s sys 0.027s
I20260504 20:00:15.188030 12940 tablet_bootstrap.cc:492] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:15.235349 12940 ts_tablet_manager.cc:1403] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.205s user 0.017s sys 0.026s
I20260504 20:00:15.237656 12944 raft_consensus.cc:359] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.238440 12944 raft_consensus.cc:385] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.238756 12944 raft_consensus.cc:740] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.239297 12944 consensus_queue.cc:260] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.244678 13041 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.247069 13041 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:15.247649 13041 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:15.196282 12645 catalog_manager.cc:5671] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:15.237215 12993 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.251349 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:15.251225 12993 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.251987 12750 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.254544 13081 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:15.255267 12897 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:15.256841 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:15.257759 12823 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.258201 12993 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.259088 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:15.262591 12716 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:00:15.263015 12791 leader_election.cc:343] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.263761 12964 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:15.263832 12992 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.264344 12964 raft_consensus.cc:697] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:15.264631 12750 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:15.264389 12992 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.265277 12964 consensus_queue.cc:237] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.266492 12992 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.267803 12865 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:15.280803 12944 ts_tablet_manager.cc:1434] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.109s user 0.007s sys 0.001s
I20260504 20:00:15.282572 12944 tablet_bootstrap.cc:492] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:15.288214 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:15.290148 13078 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 20:00:15.290340 12791 leader_election.cc:343] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.290925 13078 raft_consensus.cc:697] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:15.292212 13078 consensus_queue.cc:237] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.294030 12940 raft_consensus.cc:359] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.294914 12940 raft_consensus.cc:385] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.295244 12940 raft_consensus.cc:740] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.295912 12940 consensus_queue.cc:260] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.309594 12940 ts_tablet_manager.cc:1434] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.074s user 0.006s sys 0.002s
I20260504 20:00:15.312331 12940 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:15.312610 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:15.308490 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:15.317065 12897 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
W20260504 20:00:15.319818 12791 leader_election.cc:343] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.320451 12791 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:15.321506 12992 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:15.322208 12791 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:15.324234 12993 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:15.324718 12993 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:15.325168 12993 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.325021 12823 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:15.333788 12645 catalog_manager.cc:5671] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:15.336133 12939 tablet_bootstrap.cc:492] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:15.336566 12939 ts_tablet_manager.cc:1403] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.261s user 0.015s sys 0.018s
I20260504 20:00:15.361522 12939 raft_consensus.cc:359] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.357069 12993 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.362752 12939 raft_consensus.cc:385] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.363328 12939 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.364306 12939 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.370755 12944 tablet_bootstrap.cc:654] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.359737 12992 consensus_queue.cc:1048] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260504 20:00:15.368974 12993 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.379729 12939 ts_tablet_manager.cc:1434] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.043s user 0.006s sys 0.002s
I20260504 20:00:15.381615 12939 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:15.390933 13045 consensus_queue.cc:1048] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:15.390933 12646 catalog_manager.cc:5671] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:15.395884 12964 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:15.403347 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:15.404289 12898 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:15.408780 13082 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20260504 20:00:15.385916 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:00:15.412843 12791 leader_election.cc:343] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.424641 12940 tablet_bootstrap.cc:654] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.432073 12898 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:15.438995 12791 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:15.440266 12992 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:15.441046 12992 raft_consensus.cc:697] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:15.442525 12992 consensus_queue.cc:237] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.457875 12746 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.436450 12939 tablet_bootstrap.cc:654] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.471292 13078 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.496244 12944 tablet_bootstrap.cc:492] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:15.496784 12944 ts_tablet_manager.cc:1403] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.215s user 0.022s sys 0.019s
I20260504 20:00:15.499806 12940 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:15.500488 12940 ts_tablet_manager.cc:1403] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.188s user 0.015s sys 0.025s
I20260504 20:00:15.490391 13078 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.512462 12944 raft_consensus.cc:359] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.512641 12646 catalog_manager.cc:5671] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:15.516705 12944 raft_consensus.cc:385] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.515796 13078 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:15.517163 12944 raft_consensus.cc:740] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.518240 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:15.519786 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:15.519516 12944 consensus_queue.cc:260] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.520797 12749 raft_consensus.cc:2393] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:15.522455 12944 ts_tablet_manager.cc:1434] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20260504 20:00:15.523504 12944 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:15.525542 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:15.527740 13078 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:15.536522 12714 consensus_peers.cc:597] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Couldn't send request to peer 5ff0ef72ec3b4667893a0420d3c7f923. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 20:00:15.541447 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 43731.9us
I20260504 20:00:15.556641 12940 raft_consensus.cc:359] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.557991 12940 raft_consensus.cc:385] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.558388 12940 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.559216 12940 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.564252 12939 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:15.564877 12939 ts_tablet_manager.cc:1403] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.183s user 0.008s sys 0.013s
I20260504 20:00:15.566568 12940 ts_tablet_manager.cc:1434] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.066s user 0.006s sys 0.005s
I20260504 20:00:15.568019 13078 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:15.568358 12944 tablet_bootstrap.cc:654] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.568594 12940 tablet_bootstrap.cc:492] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:15.568980 12939 raft_consensus.cc:359] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.573983 12939 raft_consensus.cc:385] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.574373 12939 raft_consensus.cc:740] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.575621 12939 consensus_queue.cc:260] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.621313 12944 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:15.622787 12944 ts_tablet_manager.cc:1403] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.099s user 0.016s sys 0.005s
I20260504 20:00:15.634258 13078 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.636183 13078 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.644022 13078 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:15.644544 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:15.647992 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:15.649852 12940 tablet_bootstrap.cc:654] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:15.651388 12865 leader_election.cc:343] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.653117 13096 consensus_queue.cc:1048] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 20:00:15.657997 12863 leader_election.cc:343] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.660873 12865 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:15.664551 13041 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:15.668529 12950 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:15.681887 12939 ts_tablet_manager.cc:1434] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.117s user 0.002s sys 0.009s
W20260504 20:00:15.681650 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:15.686321 12939 tablet_bootstrap.cc:492] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:15.696404 13096 consensus_queue.cc:1048] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260504 20:00:15.705381 12825 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.720542 13096 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.721241 13096 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.722772 12964 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:15.725256 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:15.727002 12791 leader_election.cc:343] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.727891 12939 tablet_bootstrap.cc:654] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.728609 13096 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.731472 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:15.734529 12791 leader_election.cc:343] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.735075 12791 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:15.709822 12944 raft_consensus.cc:359] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.735858 12944 raft_consensus.cc:385] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.736056 13096 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:15.736189 12944 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.736929 12944 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.737984 12940 tablet_bootstrap.cc:492] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:15.738603 12940 ts_tablet_manager.cc:1403] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.170s user 0.042s sys 0.039s
I20260504 20:00:15.742514 12940 raft_consensus.cc:359] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.743345 12940 raft_consensus.cc:385] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.743700 12940 raft_consensus.cc:740] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.746213 12940 consensus_queue.cc:260] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.750660 12940 ts_tablet_manager.cc:1434] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260504 20:00:15.752470 12940 tablet_bootstrap.cc:492] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:15.753907 12944 ts_tablet_manager.cc:1434] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.130s user 0.009s sys 0.002s
I20260504 20:00:15.755967 12939 tablet_bootstrap.cc:492] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:15.756525 12939 ts_tablet_manager.cc:1403] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.016s
I20260504 20:00:15.759323 12746 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.761152 12939 raft_consensus.cc:359] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.761950 12939 raft_consensus.cc:385] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.762239 12939 raft_consensus.cc:740] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.762893 12939 consensus_queue.cc:260] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.772789 12944 tablet_bootstrap.cc:492] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:15.779728 12939 ts_tablet_manager.cc:1434] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20260504 20:00:15.783540 12940 tablet_bootstrap.cc:654] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.791709 12944 tablet_bootstrap.cc:654] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.792816 12939 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:15.799876 12939 tablet_bootstrap.cc:654] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.814829 13096 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.815614 13096 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.817206 12940 tablet_bootstrap.cc:492] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:15.817723 12940 ts_tablet_manager.cc:1403] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.007s
I20260504 20:00:15.824337 13096 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.824352 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:15.825721 12791 leader_election.cc:343] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.826082 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:15.826442 12940 raft_consensus.cc:359] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:15.827448 12791 leader_election.cc:343] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.827479 12940 raft_consensus.cc:385] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.828019 12791 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:15.829064 13096 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:15.828001 12940 raft_consensus.cc:740] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.830155 12940 consensus_queue.cc:260] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.832063 12944 tablet_bootstrap.cc:492] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:15.832544 12944 ts_tablet_manager.cc:1403] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.060s user 0.014s sys 0.012s
I20260504 20:00:15.832612 12939 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:15.833282 12939 ts_tablet_manager.cc:1403] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.006s
I20260504 20:00:15.833256 13033 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.835366 13033 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.836704 12944 raft_consensus.cc:359] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.837443 12944 raft_consensus.cc:385] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.837729 12944 raft_consensus.cc:740] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.838440 12944 consensus_queue.cc:260] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.839205 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:15.839475 12939 raft_consensus.cc:359] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.840361 12939 raft_consensus.cc:385] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.840471 13033 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.840654 12939 raft_consensus.cc:740] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.840775 12944 ts_tablet_manager.cc:1434] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260504 20:00:15.841589 12939 consensus_queue.cc:260] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.842293 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:15.842398 12944 tablet_bootstrap.cc:492] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:15.844455 12939 ts_tablet_manager.cc:1434] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260504 20:00:15.847390 12898 raft_consensus.cc:2393] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:15.847596 12940 ts_tablet_manager.cc:1434] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260504 20:00:15.847920 12939 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:15.852205 12716 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:15.853595 13033 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.860157 12939 tablet_bootstrap.cc:654] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.863976 13033 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:15.865295 12940 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:15.866641 12944 tablet_bootstrap.cc:654] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.873584 13041 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:15.895068 12940 tablet_bootstrap.cc:654] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.897536 13078 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:15.913367 12939 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:15.915768 12939 ts_tablet_manager.cc:1403] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.068s user 0.016s sys 0.024s
I20260504 20:00:15.917565 12944 tablet_bootstrap.cc:492] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:15.918092 12944 ts_tablet_manager.cc:1403] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.003s
I20260504 20:00:15.921659 12944 raft_consensus.cc:359] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.922689 12944 raft_consensus.cc:385] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.923074 12944 raft_consensus.cc:740] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.923908 12944 consensus_queue.cc:260] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.921226 12939 raft_consensus.cc:359] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.926156 12939 raft_consensus.cc:385] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.926467 12939 raft_consensus.cc:740] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.927191 12939 consensus_queue.cc:260] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.927948 12944 ts_tablet_manager.cc:1434] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260504 20:00:15.928368 13078 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.928855 13078 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.929497 13096 consensus_queue.cc:1048] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:15.929844 12939 ts_tablet_manager.cc:1434] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260504 20:00:15.930878 12939 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:15.932099 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:15.932771 12823 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:15.935189 12863 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:15.936146 13041 raft_consensus.cc:2804] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:15.936537 13041 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:15.936865 13041 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.937083 12944 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:15.940237 13078 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:15.927804 12940 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:15.947475 12940 ts_tablet_manager.cc:1403] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.019s
I20260504 20:00:15.946213 13041 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.948171 12939 tablet_bootstrap.cc:654] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.953406 13033 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:15.953629 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:15.954341 12750 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.954033 13033 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.954571 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:15.955229 12823 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.950979 12940 raft_consensus.cc:359] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.956146 13033 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:15.956436 12940 raft_consensus.cc:385] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:15.956743 12940 raft_consensus.cc:740] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:15.958582 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:15.960119 12714 leader_election.cc:343] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:15.961624 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:15.961687 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:15.962775 12944 tablet_bootstrap.cc:654] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:15.962622 12897 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:15.963899 13041 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:15.964046 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:15.957451 12940 consensus_queue.cc:260] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:15.965200 13033 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:15.965582 13033 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:15.965960 13033 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:15.966456 12940 ts_tablet_manager.cc:1434] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260504 20:00:15.967713 12940 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:16.046834 12939 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:16.047465 12939 ts_tablet_manager.cc:1403] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.117s user 0.011s sys 0.007s
I20260504 20:00:16.050746 12939 raft_consensus.cc:359] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.051057 12823 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:16.051759 12939 raft_consensus.cc:385] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.052219 12939 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.053071 12939 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.055992 12939 ts_tablet_manager.cc:1434] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260504 20:00:16.057274 12939 tablet_bootstrap.cc:492] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:16.059993 12750 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:16.061285 12865 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.063918 13033 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.069512 12940 tablet_bootstrap.cc:654] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.082427 13033 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.080531 13045 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.084700 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:00:16.090531 12714 leader_election.cc:343] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:16.084959 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:16.091370 12898 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.069551 13124 raft_consensus.cc:2804] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:16.094873 13124 raft_consensus.cc:697] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:16.099889 13124 consensus_queue.cc:237] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.098883 12944 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.118309 12944 ts_tablet_manager.cc:1403] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.189s user 0.012s sys 0.006s
I20260504 20:00:16.118113 13045 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.118491 12898 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.119658 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:16.110157 13119 consensus_queue.cc:1048] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:16.120476 13045 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.088442 12939 tablet_bootstrap.cc:654] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.120805 12964 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:16.121315 12964 raft_consensus.cc:697] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:16.116125 12825 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.121527 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:16.110970 12940 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:16.122949 12940 ts_tablet_manager.cc:1403] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.156s user 0.008s sys 0.020s
I20260504 20:00:16.126294 12940 raft_consensus.cc:359] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.126232 12964 consensus_queue.cc:237] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.127357 12940 raft_consensus.cc:385] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.127744 12940 raft_consensus.cc:740] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.128655 12940 consensus_queue.cc:260] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:16.129372 12791 leader_election.cc:343] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:16.131958 12940 ts_tablet_manager.cc:1434] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260504 20:00:16.133359 12940 tablet_bootstrap.cc:492] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:16.133865 12944 raft_consensus.cc:359] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.134557 12944 raft_consensus.cc:385] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.134817 12944 raft_consensus.cc:740] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.138623 12944 consensus_queue.cc:260] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.150091 12944 ts_tablet_manager.cc:1434] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20260504 20:00:16.151166 12944 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.152880 13033 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.153363 13033 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.158507 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:16.159273 12823 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.158569 12648 catalog_manager.cc:5671] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:16.160583 12714 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.161695 13097 raft_consensus.cc:2804] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:16.162187 13097 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:16.162596 13097 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.164489 12940 tablet_bootstrap.cc:654] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.154376 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.169240 12898 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:16.174911 12939 tablet_bootstrap.cc:492] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:16.175434 12939 ts_tablet_manager.cc:1403] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.118s user 0.029s sys 0.025s
I20260504 20:00:16.177451 12791 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:16.178898 13096 raft_consensus.cc:2804] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:16.178719 13097 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.179322 13096 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:16.179785 13096 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.181084 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.181468 13097 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.181775 12898 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.182289 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:16.182852 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:16.183456 12897 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.184878 12944 tablet_bootstrap.cc:654] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.179479 13033 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.179450 12646 catalog_manager.cc:5671] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:16.182896 12825 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.192713 12897 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.197431 12825 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.197865 12716 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.199084 13033 raft_consensus.cc:2804] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:16.199534 13033 raft_consensus.cc:697] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:16.200393 13033 consensus_queue.cc:237] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.204012 12939 raft_consensus.cc:359] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.205006 12944 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.196281 13096 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.209036 13096 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.205108 12939 raft_consensus.cc:385] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.210817 12939 raft_consensus.cc:740] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.211040 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:16.212196 12939 consensus_queue.cc:260] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:16.212862 12791 leader_election.cc:343] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:16.214669 12939 ts_tablet_manager.cc:1434] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.039s user 0.003s sys 0.005s
I20260504 20:00:16.215646 12939 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:16.213310 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:16.216451 12897 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.220219 12939 tablet_bootstrap.cc:654] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.207487 12944 ts_tablet_manager.cc:1403] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.001s
I20260504 20:00:16.223394 12940 tablet_bootstrap.cc:492] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:16.223937 12940 ts_tablet_manager.cc:1403] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.091s user 0.016s sys 0.006s
I20260504 20:00:16.224614 12897 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:16.224803 12944 raft_consensus.cc:359] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.225612 12944 raft_consensus.cc:385] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.225916 12944 raft_consensus.cc:740] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.225888 12791 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:16.226943 12940 raft_consensus.cc:359] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.226799 12944 consensus_queue.cc:260] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.227382 13096 raft_consensus.cc:2804] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:16.227569 12940 raft_consensus.cc:385] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.228047 13096 raft_consensus.cc:697] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:16.229368 12944 ts_tablet_manager.cc:1434] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260504 20:00:16.228938 13096 consensus_queue.cc:237] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.227941 12940 raft_consensus.cc:740] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.230423 12944 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.238401 12944 tablet_bootstrap.cc:654] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.256338 12944 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.257043 12944 ts_tablet_manager.cc:1403] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.003s
I20260504 20:00:16.260756 12944 raft_consensus.cc:359] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.261819 12944 raft_consensus.cc:385] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.262270 12944 raft_consensus.cc:740] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.263161 12944 consensus_queue.cc:260] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.266605 12944 ts_tablet_manager.cc:1434] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260504 20:00:16.268903 12944 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.270246 12940 consensus_queue.cc:260] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.268260 12646 catalog_manager.cc:5671] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:16.277534 12944 tablet_bootstrap.cc:654] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.290678 12944 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.291347 12944 ts_tablet_manager.cc:1403] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.011s
I20260504 20:00:16.292871 13123 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.293556 13123 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.295137 12944 raft_consensus.cc:359] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.296026 12944 raft_consensus.cc:385] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.296357 12944 raft_consensus.cc:740] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.298382 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:16.297269 12944 consensus_queue.cc:260] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:16.302214 12865 leader_election.cc:343] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:16.302914 13123 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:16.303020 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:16.303503 12944 ts_tablet_manager.cc:1434] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
W20260504 20:00:16.304610 12863 leader_election.cc:343] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:16.305022 12944 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.305275 12863 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:16.303772 12940 ts_tablet_manager.cc:1434] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.079s user 0.004s sys 0.003s
I20260504 20:00:16.306249 13123 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:16.308188 12940 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:16.311748 12939 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:16.312276 12939 ts_tablet_manager.cc:1403] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.097s user 0.019s sys 0.037s
I20260504 20:00:16.314594 12944 tablet_bootstrap.cc:654] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.319329 12939 raft_consensus.cc:359] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.320122 12939 raft_consensus.cc:385] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.320431 12939 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.321173 12939 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.322822 12939 ts_tablet_manager.cc:1434] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260504 20:00:16.323944 12939 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:16.326967 12944 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.327600 12944 ts_tablet_manager.cc:1403] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.007s
I20260504 20:00:16.330256 12939 tablet_bootstrap.cc:654] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.330603 12944 raft_consensus.cc:359] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.331470 12944 raft_consensus.cc:385] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.331735 12944 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.332460 12944 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.334803 12944 ts_tablet_manager.cc:1434] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260504 20:00:16.335794 12944 tablet_bootstrap.cc:492] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.338465 12940 tablet_bootstrap.cc:654] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.344897 12944 tablet_bootstrap.cc:654] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.351649 12940 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:16.353395 12940 ts_tablet_manager.cc:1403] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.002s
I20260504 20:00:16.354982 12646 catalog_manager.cc:5671] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:16.356177 12940 raft_consensus.cc:359] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.356866 12940 raft_consensus.cc:385] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.357147 12940 raft_consensus.cc:740] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.357828 12940 consensus_queue.cc:260] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.360003 12940 ts_tablet_manager.cc:1434] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260504 20:00:16.361538 12940 tablet_bootstrap.cc:492] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:16.364518 12944 tablet_bootstrap.cc:492] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.365027 12944 ts_tablet_manager.cc:1403] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.017s
I20260504 20:00:16.367972 12944 raft_consensus.cc:359] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.368912 12944 raft_consensus.cc:385] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.369204 12944 raft_consensus.cc:740] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.369789 12944 consensus_queue.cc:260] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.372450 12944 ts_tablet_manager.cc:1434] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260504 20:00:16.373500 12944 tablet_bootstrap.cc:492] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.379575 12944 tablet_bootstrap.cc:654] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.395028 12940 tablet_bootstrap.cc:654] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.401239 12939 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:16.401521 12940 tablet_bootstrap.cc:492] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:16.401742 12939 ts_tablet_manager.cc:1403] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.078s user 0.028s sys 0.033s
I20260504 20:00:16.402033 12940 ts_tablet_manager.cc:1403] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.000s
I20260504 20:00:16.404585 12939 raft_consensus.cc:359] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.405393 12939 raft_consensus.cc:385] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.405675 12939 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.407658 12939 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.408094 12940 raft_consensus.cc:359] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.408823 12940 raft_consensus.cc:385] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.409111 12940 raft_consensus.cc:740] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.409835 12940 consensus_queue.cc:260] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.410755 12939 ts_tablet_manager.cc:1434] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260504 20:00:16.411401 12940 ts_tablet_manager.cc:1434] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260504 20:00:16.411754 12939 tablet_bootstrap.cc:492] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:16.412688 12940 tablet_bootstrap.cc:492] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:16.411688 12944 tablet_bootstrap.cc:492] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.416990 12944 ts_tablet_manager.cc:1403] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.044s user 0.025s sys 0.006s
I20260504 20:00:16.421867 12944 raft_consensus.cc:359] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.422574 12944 raft_consensus.cc:385] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.422848 12944 raft_consensus.cc:740] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.423501 12944 consensus_queue.cc:260] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.424449 12939 tablet_bootstrap.cc:654] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.425513 12944 ts_tablet_manager.cc:1434] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20260504 20:00:16.421870 12940 tablet_bootstrap.cc:654] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.429524 12944 tablet_bootstrap.cc:492] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.435745 12944 tablet_bootstrap.cc:654] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.437430 12940 tablet_bootstrap.cc:492] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:16.437953 12940 ts_tablet_manager.cc:1403] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.000s
I20260504 20:00:16.441191 12940 raft_consensus.cc:359] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.442157 12940 raft_consensus.cc:385] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.442509 12940 raft_consensus.cc:740] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.443351 12940 consensus_queue.cc:260] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.446740 12939 tablet_bootstrap.cc:492] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:16.447301 12944 tablet_bootstrap.cc:492] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.447404 12939 ts_tablet_manager.cc:1403] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.013s
I20260504 20:00:16.447438 12940 ts_tablet_manager.cc:1434] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260504 20:00:16.447772 12944 ts_tablet_manager.cc:1403] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260504 20:00:16.449266 12940 tablet_bootstrap.cc:492] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:16.450430 12939 raft_consensus.cc:359] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.451128 12939 raft_consensus.cc:385] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.450866 12944 raft_consensus.cc:359] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.451419 12939 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.451553 12944 raft_consensus.cc:385] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.451876 12944 raft_consensus.cc:740] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.452111 12939 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.452595 12944 consensus_queue.cc:260] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.454371 12939 ts_tablet_manager.cc:1434] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260504 20:00:16.454663 12944 ts_tablet_manager.cc:1434] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260504 20:00:16.455629 12939 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:16.455638 12944 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.459213 12940 tablet_bootstrap.cc:654] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.462007 12939 tablet_bootstrap.cc:654] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.464146 12944 tablet_bootstrap.cc:654] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.473204 13033 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.473948 13033 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.483460 13097 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.484236 13097 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.490229 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:16.491144 12825 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.492254 13119 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.492748 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.492828 13119 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.495802 13119 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.497033 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.499823 12964 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:16.500379 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:16.501359 13097 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.500594 12964 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:16.502259 12964 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.507513 13097 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.508199 13097 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.510043 12897 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.511837 13033 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.501298 12750 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:16.514075 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:16.514935 12825 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.515214 12791 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.516356 12714 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.517503 13045 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:16.518051 13045 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:16.518478 13045 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.518682 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.519618 12898 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:16.521592 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.522698 12897 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.531352 12950 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:16.531960 12950 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:16.532414 12950 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.537698 13097 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.539465 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:16.540423 12823 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.542316 12714 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.543473 13033 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:16.543941 13033 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:16.548658 13033 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.554361 12964 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.566582 13122 consensus_queue.cc:1048] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:16.598317 13123 consensus_queue.cc:1048] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:16.600278 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.601246 12897 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.611760 13045 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.617700 13155 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.620877 13155 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.623553 13155 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.580338 12950 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.627082 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:16.627903 12823 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.625514 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:16.630241 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.631456 12898 raft_consensus.cc:2393] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:16.633379 13033 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.638437 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:16.648942 12898 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.648833 13156 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:16.656137 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:16.657191 12825 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.639712 13045 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.642333 13097 consensus_queue.cc:1048] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:16.626602 12950 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.643599 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:16.659756 12749 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.661933 12964 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.664916 12823 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.669108 12749 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:16.670473 12944 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.669510 12939 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:16.671209 12939 ts_tablet_manager.cc:1403] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.216s user 0.090s sys 0.106s
I20260504 20:00:16.671300 12944 ts_tablet_manager.cc:1403] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.216s user 0.092s sys 0.103s
I20260504 20:00:16.671459 12791 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.672967 13120 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:16.673442 13120 raft_consensus.cc:697] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:16.674911 12940 tablet_bootstrap.cc:492] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:16.674320 13120 consensus_queue.cc:237] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.675429 12940 ts_tablet_manager.cc:1403] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.226s user 0.019s sys 0.017s
I20260504 20:00:16.674324 12944 raft_consensus.cc:359] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.678180 12944 raft_consensus.cc:385] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.678038 12940 raft_consensus.cc:359] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.678519 12944 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.678767 12940 raft_consensus.cc:385] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.679180 12940 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.679522 12944 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.680557 12939 raft_consensus.cc:359] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.681296 12939 raft_consensus.cc:385] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.647476 12716 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:16.681602 12939 raft_consensus.cc:740] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.683501 12939 consensus_queue.cc:260] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.683800 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:16.684392 12950 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.682114 12940 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.684459 12822 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.682514 12944 ts_tablet_manager.cc:1434] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260504 20:00:16.685777 12939 ts_tablet_manager.cc:1434] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260504 20:00:16.687908 12939 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:16.690992 13033 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.693964 12822 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.695011 12944 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:16.698433 12939 tablet_bootstrap.cc:654] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.729594 12714 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.730696 12940 ts_tablet_manager.cc:1434] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20260504 20:00:16.732227 13156 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:16.732636 12940 tablet_bootstrap.cc:492] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:16.732750 13156 raft_consensus.cc:697] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:16.716010 13118 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.733892 13156 consensus_queue.cc:237] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.732790 12714 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.735634 13097 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:16.736136 13097 raft_consensus.cc:697] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:16.749155 12825 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.765323 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:16.757493 13033 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.737509 13097 consensus_queue.cc:237] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.766966 12897 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.767266 12964 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.733402 13118 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.771756 12964 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.770124 13033 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.783524 13033 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.783881 12964 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.770349 12898 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.774061 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.787469 13033 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:16.788851 13033 raft_consensus.cc:697] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:16.789749 13033 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:16.791143 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:16.795030 12950 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:16.806306 12944 tablet_bootstrap.cc:654] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.809458 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:16.810148 12899 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.811172 13120 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:16.813011 13118 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.801695 12897 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.846104 12964 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.860423 12964 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.862284 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.872872 12939 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:16.873320 12939 ts_tablet_manager.cc:1403] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.186s user 0.025s sys 0.021s
I20260504 20:00:16.868227 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:16.880625 12749 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:16.896627 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.896533 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:16.882083 12964 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:16.898334 13120 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:16.901948 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:16.902438 12940 tablet_bootstrap.cc:654] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:16.904650 13045 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:16.911795 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:16.912276 12714 consensus_peers.cc:597] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Couldn't send request to peer 5ff0ef72ec3b4667893a0420d3c7f923. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:16.912801 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:16.913533 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:16.914279 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:16.915736 12944 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:16.916988 12899 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:16.919983 12944 ts_tablet_manager.cc:1403] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.225s user 0.022s sys 0.012s
I20260504 20:00:16.921232 12648 catalog_manager.cc:5671] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:16.903458 13120 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.927067 12896 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.915867 13045 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:00:16.930044 12714 leader_election.cc:343] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:16.927315 12822 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:16.934856 12939 raft_consensus.cc:359] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.948987 12939 raft_consensus.cc:385] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:16.949431 12939 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:16.950311 12939 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:16.944564 12964 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260504 20:00:16.948807 13045 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.973081 13120 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:16.955677 12716 leader_election.cc:343] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:16.980012 12939 ts_tablet_manager.cc:1434] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.106s user 0.014s sys 0.004s
I20260504 20:00:16.981258 12939 tablet_bootstrap.cc:492] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:16.981053 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:16.983542 12714 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
W20260504 20:00:16.986243 12791 leader_election.cc:343] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:16.990906 12964 raft_consensus.cc:2804] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:16.991351 12964 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:16.991690 12964 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:16.998584 13045 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.010301 12944 raft_consensus.cc:359] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.013906 12944 raft_consensus.cc:385] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 20:00:16.999944 12716 leader_election.cc:343] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:16.999982 13162 log.cc:927] Time spent T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260504 20:00:17.005419 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:17.019402 12899 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:17.019795 12939 tablet_bootstrap.cc:654] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:17.007699 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:17.008291 12645 catalog_manager.cc:5671] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:17.024015 12645 catalog_manager.cc:5671] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:17.025871 13045 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:17.010268 12964 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.012365 12896 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:17.015278 12823 raft_consensus.cc:2393] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:17.015643 12944 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.034700 12944 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.037611 13120 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:17.038774 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:17.037616 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:17.039950 12749 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:17.042651 12716 leader_election.cc:343] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.041404 12964 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:17.043308 12716 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:17.043493 12940 tablet_bootstrap.cc:492] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:17.048797 12944 ts_tablet_manager.cc:1434] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.126s user 0.006s sys 0.000s
I20260504 20:00:17.048633 13156 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:17.049540 12940 ts_tablet_manager.cc:1403] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.317s user 0.019s sys 0.022s
I20260504 20:00:17.051457 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:17.052156 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:17.053478 12823 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:17.057060 12791 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:17.063463 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:17.066011 13121 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:17.066938 13121 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:17.067489 13121 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:17.067859 12940 raft_consensus.cc:359] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.068739 12940 raft_consensus.cc:385] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.069031 12940 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.069702 12940 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.072032 12823 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:17.072558 12940 ts_tablet_manager.cc:1434] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260504 20:00:17.074533 12940 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:17.074736 12714 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:17.075850 13156 raft_consensus.cc:2804] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:17.076521 13156 raft_consensus.cc:697] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:17.077692 13156 consensus_queue.cc:237] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:17.084733 12791 leader_election.cc:343] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.086400 12944 tablet_bootstrap.cc:492] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:17.091122 12749 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:17.094614 13097 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:17.095590 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:17.097170 13096 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:17.097951 13096 raft_consensus.cc:697] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:17.099366 13096 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:17.102716 12716 leader_election.cc:343] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.124238 13118 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:17.130118 13118 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.129642 12939 tablet_bootstrap.cc:492] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:17.133023 12939 ts_tablet_manager.cc:1403] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.152s user 0.018s sys 0.017s
I20260504 20:00:17.138217 13120 consensus_queue.cc:1048] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:17.142204 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:17.143074 12939 raft_consensus.cc:359] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.143625 12939 raft_consensus.cc:385] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.143853 12939 raft_consensus.cc:740] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.138957 13118 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:17.148581 12940 tablet_bootstrap.cc:654] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:17.144383 12939 consensus_queue.cc:260] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:17.157290 12791 leader_election.cc:343] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.159193 12646 catalog_manager.cc:5671] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:17.161744 12646 catalog_manager.cc:5671] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:17.167398 12939 ts_tablet_manager.cc:1434] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.034s user 0.000s sys 0.007s
I20260504 20:00:17.185948 12939 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:17.190251 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:17.203506 12823 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:17.192469 13120 consensus_queue.cc:1048] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260504 20:00:17.211447 12791 leader_election.cc:343] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.212141 12791 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:17.199718 13121 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.233584 12944 tablet_bootstrap.cc:654] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:17.216239 13120 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:17.233770 12645 catalog_manager.cc:5671] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:17.237321 13123 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:17.237860 12940 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:17.238358 12940 ts_tablet_manager.cc:1403] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.164s user 0.012s sys 0.026s
I20260504 20:00:17.238027 13123 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.242552 13123 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:17.242835 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:17.246642 12863 leader_election.cc:343] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.252864 13097 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260504 20:00:17.254869 13121 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:17.255843 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:00:17.258260 12791 leader_election.cc:343] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.278405 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:17.280556 12939 tablet_bootstrap.cc:654] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:17.309232 12940 raft_consensus.cc:359] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.311256 12940 raft_consensus.cc:385] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.311594 12940 raft_consensus.cc:740] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.312065 12940 consensus_queue.cc:260] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.317755 12940 ts_tablet_manager.cc:1434] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.079s user 0.009s sys 0.001s
W20260504 20:00:17.319523 12865 leader_election.cc:343] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.319872 12944 tablet_bootstrap.cc:492] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:17.320107 12865 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:17.320407 12940 tablet_bootstrap.cc:492] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:17.321094 13123 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:17.324754 12944 ts_tablet_manager.cc:1403] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.241s user 0.014s sys 0.027s
I20260504 20:00:17.323271 12964 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260504 20:00:17.327215 13156 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260504 20:00:17.337167 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:17.337978 12894 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:17.348678 13097 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:17.379087 12944 raft_consensus.cc:359] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.379969 12944 raft_consensus.cc:385] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.380303 12944 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.380980 12944 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.383450 12944 ts_tablet_manager.cc:1434] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20260504 20:00:17.388665 12939 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:17.389194 12939 ts_tablet_manager.cc:1403] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.220s user 0.036s sys 0.023s
I20260504 20:00:17.392431 13155 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:17.394416 12944 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:17.408402 13033 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:17.419915 12894 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:17.430831 12939 raft_consensus.cc:359] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.433148 12939 raft_consensus.cc:385] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.433535 12939 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.433775 12791 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:17.434454 12939 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.435144 13121 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:17.436295 13121 raft_consensus.cc:697] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:17.437678 13121 consensus_queue.cc:237] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.439291 12939 ts_tablet_manager.cc:1434] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.050s user 0.009s sys 0.001s
I20260504 20:00:17.443442 12939 tablet_bootstrap.cc:492] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:17.447165 13156 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:17.448069 13156 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.455768 13123 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:17.456640 13123 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.489517 12940 tablet_bootstrap.cc:654] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:17.496729 13187 log.cc:927] Time spent T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260504 20:00:17.516791 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:17.518518 13156 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:17.538960 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:17.540994 12944 tablet_bootstrap.cc:654] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:17.537858 13123 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:17.533972 13096 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260504 20:00:17.562441 13097 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:17.545922 12820 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:17.569092 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:17.586064 12939 tablet_bootstrap.cc:654] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:17.631050 12865 leader_election.cc:343] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.636035 12940 tablet_bootstrap.cc:492] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:17.636623 12940 ts_tablet_manager.cc:1403] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.316s user 0.014s sys 0.018s
I20260504 20:00:17.592623 12645 catalog_manager.cc:5671] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:00:17.638253 12863 leader_election.cc:343] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.642364 12863 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:17.653426 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:17.659998 13196 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:17.682202 12940 raft_consensus.cc:359] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.683099 12940 raft_consensus.cc:385] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.683445 12940 raft_consensus.cc:740] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.684221 12940 consensus_queue.cc:260] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.693243 12940 ts_tablet_manager.cc:1434] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.056s user 0.006s sys 0.002s
I20260504 20:00:17.683465 12714 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:17.694578 12940 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:17.694509 13033 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:17.695170 13033 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:17.695652 13033 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:17.706799 12716 leader_election.cc:343] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.712332 13097 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:17.713220 13097 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.722309 12939 tablet_bootstrap.cc:492] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:17.723017 12939 ts_tablet_manager.cc:1403] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.280s user 0.019s sys 0.022s
W20260504 20:00:17.723663 12791 consensus_peers.cc:597] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Couldn't send request to peer 732c3560a688494196d22a1a48f89968. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:17.724546 13097 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:17.740204 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:17.747715 12939 raft_consensus.cc:359] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.751576 12939 raft_consensus.cc:385] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.751961 12939 raft_consensus.cc:740] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.750347 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:17.757855 12896 raft_consensus.cc:2393] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:17.765722 12716 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:17.773423 13197 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:17.775729 13202 log.cc:927] Time spent T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260504 20:00:17.753139 12939 consensus_queue.cc:260] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.791417 12939 ts_tablet_manager.cc:1434] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.068s user 0.010s sys 0.000s
I20260504 20:00:17.792869 12939 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
W20260504 20:00:17.796828 12716 consensus_peers.cc:597] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Couldn't send request to peer 732c3560a688494196d22a1a48f89968. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:17.798913 12940 tablet_bootstrap.cc:654] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:17.805760 13197 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:17.809146 13033 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.812683 13033 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:17.823027 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:00:17.825531 12716 leader_election.cc:343] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:17.831640 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:17.832665 12819 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:17.843338 12940 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:17.843346 12944 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:17.843955 12940 ts_tablet_manager.cc:1403] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.150s user 0.012s sys 0.009s
I20260504 20:00:17.844278 12944 ts_tablet_manager.cc:1403] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.450s user 0.028s sys 0.092s
I20260504 20:00:17.847059 12819 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:17.848522 12944 raft_consensus.cc:359] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.848762 12714 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:17.850870 13156 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:17.851613 13156 raft_consensus.cc:697] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:17.853278 12940 raft_consensus.cc:359] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.854013 12940 raft_consensus.cc:385] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.854276 12940 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.852932 13156 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.854898 12940 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.861150 12940 ts_tablet_manager.cc:1434] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.017s user 0.004s sys 0.003s
I20260504 20:00:17.856106 12939 tablet_bootstrap.cc:654] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:17.860859 13119 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:17.861610 12944 raft_consensus.cc:385] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:17.862147 12944 raft_consensus.cc:740] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:17.866501 12940 tablet_bootstrap.cc:492] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:17.863422 12944 consensus_queue.cc:260] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:17.875345 12944 ts_tablet_manager.cc:1434] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20260504 20:00:17.875592 12940 tablet_bootstrap.cc:654] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:17.876448 12944 tablet_bootstrap.cc:492] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:17.903302 13196 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:17.904946 13196 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.008838 13123 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.010608 13123 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.014842 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:18.015909 12749 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:18.024124 12865 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:18.061079 13198 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:18.061630 13198 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:18.062080 13198 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.063194 13196 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:18.065099 13123 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:18.115381 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 91354.3us
I20260504 20:00:18.145046 13198 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.165560 13198 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:18.170104 12939 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:18.170680 12939 ts_tablet_manager.cc:1403] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.378s user 0.021s sys 0.008s
I20260504 20:00:18.176043 12940 tablet_bootstrap.cc:492] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:18.176589 12940 ts_tablet_manager.cc:1403] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.310s user 0.008s sys 0.009s
I20260504 20:00:18.180130 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:00:18.180550 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:18.180789 12746 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.181035 12645 catalog_manager.cc:5671] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:18.181712 12939 raft_consensus.cc:359] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.183246 12939 raft_consensus.cc:385] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.183552 12939 raft_consensus.cc:740] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.184166 12939 consensus_queue.cc:260] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.188066 12939 ts_tablet_manager.cc:1434] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260504 20:00:18.191390 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:18.232493 13198 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.233171 13198 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.243536 13198 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:18.263396 12944 tablet_bootstrap.cc:654] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.267000 12940 raft_consensus.cc:359] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.272601 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:18.274196 13033 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.297757 12940 raft_consensus.cc:385] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.298185 12940 raft_consensus.cc:740] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.298125 13033 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.298835 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:18.299052 12940 consensus_queue.cc:260] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.300352 13033 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:18.274883 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:18.275830 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 90219.1us
W20260504 20:00:18.302568 12863 leader_election.cc:343] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:18.277446 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:18.303740 12824 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:18.303578 13096 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:18.280346 12746 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:18.285629 13197 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260504 20:00:18.275790 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:18.287366 12939 tablet_bootstrap.cc:492] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:18.310459 12865 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:18.313851 12940 ts_tablet_manager.cc:1434] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.137s user 0.044s sys 0.034s
I20260504 20:00:18.320907 13123 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:18.320950 12863 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:18.330914 13198 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:18.331552 13198 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:18.332671 12940 tablet_bootstrap.cc:492] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:18.332098 13198 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:18.332686 12865 leader_election.cc:343] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:18.335471 12863 leader_election.cc:343] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:18.341507 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:18.342911 12819 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:18.347379 12714 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:18.352275 12944 tablet_bootstrap.cc:492] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:18.352910 12944 ts_tablet_manager.cc:1403] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.477s user 0.010s sys 0.016s
I20260504 20:00:18.355278 13197 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:18.357259 13197 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:18.357728 13197 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.359089 12944 raft_consensus.cc:359] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.359360 12865 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:18.359817 12944 raft_consensus.cc:385] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.361281 13208 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:18.361934 13208 raft_consensus.cc:697] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:18.360611 12944 raft_consensus.cc:740] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.363250 13208 consensus_queue.cc:237] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.363891 12944 consensus_queue.cc:260] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.369820 12895 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.370842 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:18.371644 12894 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:18.374185 12944 ts_tablet_manager.cc:1434] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260504 20:00:18.383944 12944 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:18.369323 13198 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.412552 13214 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.413468 13214 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.400040 12646 catalog_manager.cc:5671] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:18.442308 13214 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:18.468784 13197 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.490713 13197 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:18.492399 12940 tablet_bootstrap.cc:654] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.494819 12939 tablet_bootstrap.cc:654] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.501155 12944 tablet_bootstrap.cc:654] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.500200 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:18.503998 12824 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:18.457306 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:18.514154 13219 log.cc:927] Time spent T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260504 20:00:18.516786 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:18.518265 12748 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.519251 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:18.520121 12823 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.517113 13198 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:18.510349 12716 consensus_peers.cc:597] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Couldn't send request to peer 732c3560a688494196d22a1a48f89968. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:18.533149 12893 raft_consensus.cc:2393] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:18.541158 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:18.542032 12824 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.529413 12714 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:18.547664 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:18.548594 12893 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.568145 12824 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:18.581686 12714 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:18.583796 13214 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:18.570758 12823 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:18.584529 13214 raft_consensus.cc:697] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:18.588274 12940 tablet_bootstrap.cc:492] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:18.588930 12940 ts_tablet_manager.cc:1403] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.264s user 0.023s sys 0.044s
I20260504 20:00:18.590884 12939 tablet_bootstrap.cc:492] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:18.591380 12939 ts_tablet_manager.cc:1403] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.304s user 0.044s sys 0.031s
I20260504 20:00:18.593313 12940 raft_consensus.cc:359] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.593071 13214 consensus_queue.cc:237] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.594105 12940 raft_consensus.cc:385] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.594434 12940 raft_consensus.cc:740] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.594650 12939 raft_consensus.cc:359] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.595309 12939 raft_consensus.cc:385] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.595118 12940 consensus_queue.cc:260] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.595567 12939 raft_consensus.cc:740] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.597465 12940 ts_tablet_manager.cc:1434] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260504 20:00:18.597677 12939 consensus_queue.cc:260] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.598838 12940 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:18.600414 12939 ts_tablet_manager.cc:1434] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260504 20:00:18.600523 12748 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:18.605698 12939 tablet_bootstrap.cc:492] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:18.605487 12865 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:18.606385 12964 raft_consensus.cc:2764] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260504 20:00:18.606980 13198 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:18.607950 13198 raft_consensus.cc:697] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:18.619253 12893 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:18.620627 13198 consensus_queue.cc:237] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.629506 12944 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:18.630106 12944 ts_tablet_manager.cc:1403] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.246s user 0.010s sys 0.028s
I20260504 20:00:18.632555 12940 tablet_bootstrap.cc:654] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.633385 12944 raft_consensus.cc:359] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.634095 12944 raft_consensus.cc:385] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.634352 12944 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.635022 12944 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.637442 12939 tablet_bootstrap.cc:654] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.637583 12944 ts_tablet_manager.cc:1434] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260504 20:00:18.640841 12944 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:18.643460 12645 catalog_manager.cc:5671] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:18.652696 12944 tablet_bootstrap.cc:654] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.679966 12944 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:18.680476 12944 ts_tablet_manager.cc:1403] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.010s
I20260504 20:00:18.684800 12645 catalog_manager.cc:5671] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:18.690212 12940 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:18.690754 12940 ts_tablet_manager.cc:1403] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.030s
I20260504 20:00:18.691116 12944 raft_consensus.cc:359] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.692209 12944 raft_consensus.cc:385] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.692466 12944 raft_consensus.cc:740] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.693619 12944 consensus_queue.cc:260] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.695984 12940 raft_consensus.cc:359] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.696696 12940 raft_consensus.cc:385] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.696950 12940 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.697649 12940 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.699420 12939 tablet_bootstrap.cc:492] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:18.700448 12939 ts_tablet_manager.cc:1403] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.095s user 0.027s sys 0.016s
I20260504 20:00:18.700642 13198 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.700897 12944 ts_tablet_manager.cc:1434] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.020s user 0.009s sys 0.001s
I20260504 20:00:18.701282 13198 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.702255 12944 tablet_bootstrap.cc:492] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:18.704002 12939 raft_consensus.cc:359] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.704653 12939 raft_consensus.cc:385] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.704996 12939 raft_consensus.cc:740] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.705164 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:18.705287 13198 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:18.705873 12939 consensus_queue.cc:260] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.706303 12940 ts_tablet_manager.cc:1434] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.015s user 0.006s sys 0.004s
W20260504 20:00:18.707931 12865 leader_election.cc:343] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:18.708062 13216 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.708571 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:18.708705 13216 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:18.713951 12863 leader_election.cc:343] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:18.714570 12863 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:18.716179 13123 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:18.716307 13216 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:18.707930 12940 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:18.720925 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:18.726604 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:18.727367 12944 tablet_bootstrap.cc:654] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.730515 12750 raft_consensus.cc:2393] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:18.735787 12791 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:18.737625 12939 ts_tablet_manager.cc:1434] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260504 20:00:18.739082 13211 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.743753 12939 tablet_bootstrap.cc:492] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:18.757289 13211 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:18.768079 12940 tablet_bootstrap.cc:654] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.782423 12939 tablet_bootstrap.cc:654] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.796353 12944 tablet_bootstrap.cc:492] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:18.796897 12944 ts_tablet_manager.cc:1403] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.095s user 0.013s sys 0.030s
I20260504 20:00:18.798230 12940 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:18.798754 12940 ts_tablet_manager.cc:1403] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.011s
I20260504 20:00:18.800793 12939 tablet_bootstrap.cc:492] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:18.800487 12944 raft_consensus.cc:359] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.801250 12944 raft_consensus.cc:385] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.801339 12939 ts_tablet_manager.cc:1403] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.012s
I20260504 20:00:18.801532 12944 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.802223 12940 raft_consensus.cc:359] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.802282 12944 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.802930 12940 raft_consensus.cc:385] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.803196 12940 raft_consensus.cc:740] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.803843 12940 consensus_queue.cc:260] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.804497 12944 ts_tablet_manager.cc:1434] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260504 20:00:18.806869 12940 ts_tablet_manager.cc:1434] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260504 20:00:18.828591 12944 tablet_bootstrap.cc:492] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:18.831954 12940 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:18.837787 13198 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.838498 13198 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.850056 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:18.850622 13198 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:18.851055 12748 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:18.851516 12940 tablet_bootstrap.cc:654] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.852070 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:18.852463 12865 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:18.852734 12823 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:18.853569 13123 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:18.853837 12944 tablet_bootstrap.cc:654] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.853966 13123 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:18.854470 13123 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.850019 12939 raft_consensus.cc:359] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.858270 12964 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.859455 12939 raft_consensus.cc:385] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.859540 12964 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.859889 12939 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.860682 12939 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.863948 12939 ts_tablet_manager.cc:1434] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.062s user 0.005s sys 0.005s
I20260504 20:00:18.877324 12964 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:18.882767 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:18.883853 12823 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:18.885679 12714 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:18.885751 13198 consensus_queue.cc:1048] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:18.887847 13214 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:18.888403 13214 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:18.888900 13214 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.892225 12944 tablet_bootstrap.cc:492] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:18.892838 12944 ts_tablet_manager.cc:1403] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.065s user 0.015s sys 0.011s
I20260504 20:00:18.893466 12939 tablet_bootstrap.cc:492] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:18.895531 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:18.899878 12893 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:18.897646 13123 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.915684 12940 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:18.916216 12940 ts_tablet_manager.cc:1403] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.085s user 0.022s sys 0.030s
I20260504 20:00:18.917647 13120 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:18.918792 13120 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.921991 13120 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:18.926390 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:18.927345 12893 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:18.934783 13123 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:18.938563 12944 raft_consensus.cc:359] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.939613 12944 raft_consensus.cc:385] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.921998 13214 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.946557 13214 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:18.949117 12944 raft_consensus.cc:740] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.963837 12944 consensus_queue.cc:260] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.967406 12944 ts_tablet_manager.cc:1434] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.074s user 0.007s sys 0.007s
I20260504 20:00:18.968744 12944 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:18.965237 12940 raft_consensus.cc:359] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.969529 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:18.970268 12940 raft_consensus.cc:385] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:18.970587 12940 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:18.970664 12748 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.970933 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:18.971539 12750 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:18.971489 12940 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:18.972826 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:18.973469 12824 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.974113 12940 ts_tablet_manager.cc:1434] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.058s user 0.010s sys 0.003s
I20260504 20:00:18.975603 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:18.976307 12823 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.970000 12791 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:18.969997 12939 tablet_bootstrap.cc:654] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:18.978572 12895 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.978781 13120 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:18.979362 13120 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:18.979880 13120 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.988989 12940 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:18.991380 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:18.992323 12899 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:18.998636 12824 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:19.012164 12748 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:19.012748 12863 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:19.014613 12823 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:19.016607 13122 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:19.017427 13122 raft_consensus.cc:697] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:19.018895 13122 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.017333 13120 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.027977 12714 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:19.032073 12964 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:19.032680 12964 raft_consensus.cc:697] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:19.033532 12964 consensus_queue.cc:237] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.028281 13214 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:19.034796 12944 tablet_bootstrap.cc:654] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.064666 12899 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:19.087011 12940 tablet_bootstrap.cc:654] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.107225 12939 tablet_bootstrap.cc:492] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:19.108701 12939 ts_tablet_manager.cc:1403] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.237s user 0.021s sys 0.021s
I20260504 20:00:19.117291 13120 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:19.120143 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:19.120545 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:19.124673 12748 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.127517 12899 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.112872 13214 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:19.134462 13214 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.137190 13214 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:19.141335 12748 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
W20260504 20:00:19.108997 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 49712.3us
I20260504 20:00:19.146175 12791 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:19.151911 13120 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:19.154377 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:19.160584 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:19.154528 13120 raft_consensus.cc:697] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:00:19.163200 12716 leader_election.cc:343] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.163806 13120 consensus_queue.cc:237] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.169328 12899 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
W20260504 20:00:19.171892 12714 leader_election.cc:343] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.172418 12714 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:19.173242 13243 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:19.161130 12644 catalog_manager.cc:5671] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:19.177352 12940 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:19.177945 12940 ts_tablet_manager.cc:1403] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.200s user 0.023s sys 0.014s
I20260504 20:00:19.182518 12939 raft_consensus.cc:359] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.184552 12939 raft_consensus.cc:385] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.185075 12939 raft_consensus.cc:740] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.190270 12939 consensus_queue.cc:260] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.183286 13122 consensus_queue.cc:1048] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260504 20:00:19.201136 12940 raft_consensus.cc:359] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.203266 12940 raft_consensus.cc:385] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.203580 12940 raft_consensus.cc:740] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.205304 12940 consensus_queue.cc:260] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.216511 12646 catalog_manager.cc:5671] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:19.217934 12893 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.218564 12939 ts_tablet_manager.cc:1434] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.109s user 0.008s sys 0.000s
I20260504 20:00:19.234107 12940 ts_tablet_manager.cc:1434] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.056s user 0.006s sys 0.002s
I20260504 20:00:19.202102 13197 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260504 20:00:19.235077 12940 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:19.234617 12939 tablet_bootstrap.cc:492] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:19.246904 13122 consensus_queue.cc:1048] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:19.251331 12940 tablet_bootstrap.cc:654] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.255203 12939 tablet_bootstrap.cc:654] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.260141 12944 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:19.267553 12944 ts_tablet_manager.cc:1403] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.299s user 0.026s sys 0.031s
I20260504 20:00:19.271893 12939 tablet_bootstrap.cc:492] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:19.289145 12939 ts_tablet_manager.cc:1403] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20260504 20:00:19.290082 13240 consensus_queue.cc:1048] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:19.291975 12939 raft_consensus.cc:359] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.292721 12939 raft_consensus.cc:385] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.292968 12939 raft_consensus.cc:740] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.293079 12940 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:19.294400 12940 ts_tablet_manager.cc:1403] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.012s
I20260504 20:00:19.293839 12939 consensus_queue.cc:260] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:19.296157 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:19.297497 12939 ts_tablet_manager.cc:1434] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260504 20:00:19.298597 12939 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:19.298424 12940 raft_consensus.cc:359] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.299010 12940 raft_consensus.cc:385] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.299338 12940 raft_consensus.cc:740] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.299997 12940 consensus_queue.cc:260] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.307888 12939 tablet_bootstrap.cc:654] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.308248 12940 ts_tablet_manager.cc:1434] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260504 20:00:19.310479 12944 raft_consensus.cc:359] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.311499 12944 raft_consensus.cc:385] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.311852 12944 raft_consensus.cc:740] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.312669 12944 consensus_queue.cc:260] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.335094 12944 ts_tablet_manager.cc:1434] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20260504 20:00:19.338474 12940 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:19.337957 13121 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:19.341125 13121 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.343212 13121 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:19.345343 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:19.346269 12899 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:19.347684 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:19.345515 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:19.349628 13120 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:19.350029 13120 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:19.350986 13214 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:19.351428 13120 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.351799 13214 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.363871 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:19.364629 12944 tablet_bootstrap.cc:492] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:19.365003 12895 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
W20260504 20:00:19.364208 12791 leader_election.cc:343] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.366291 13214 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:19.368223 12716 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:19.375196 13197 raft_consensus.cc:2804] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:19.375828 13197 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:19.376358 13197 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:19.453712 13251 log.cc:927] Time spent T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.063s user 0.005s sys 0.000s
W20260504 20:00:19.454020 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 54065.5us
I20260504 20:00:19.389192 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:19.453405 13258 log.cc:927] Time spent T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260504 20:00:19.462522 12940 tablet_bootstrap.cc:654] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.462189 13242 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:19.464375 13240 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:19.465968 13242 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.463843 12646 catalog_manager.cc:5671] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
W20260504 20:00:19.472863 12714 leader_election.cc:343] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.466109 13240 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.537503 13123 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:19.559943 13240 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:19.561483 12944 tablet_bootstrap.cc:654] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.563437 13242 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:19.564517 13120 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.566505 13120 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:19.571095 12944 tablet_bootstrap.cc:492] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:19.612726 12944 ts_tablet_manager.cc:1403] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.274s user 0.014s sys 0.038s
I20260504 20:00:19.612926 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:19.613737 12940 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:19.614416 12940 ts_tablet_manager.cc:1403] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.280s user 0.016s sys 0.009s
I20260504 20:00:19.573657 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:19.574324 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:19.581636 12939 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:19.593217 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:19.617975 12750 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:19.595108 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:19.595296 13197 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.606858 13261 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:19.621749 13261 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.622296 12944 raft_consensus.cc:359] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.623143 12944 raft_consensus.cc:385] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.623484 12944 raft_consensus.cc:740] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.623898 13240 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:19.624241 12944 consensus_queue.cc:260] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.624794 13261 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:19.618515 12863 leader_election.cc:343] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.626004 12939 ts_tablet_manager.cc:1403] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.328s user 0.050s sys 0.060s
I20260504 20:00:19.629927 12939 raft_consensus.cc:359] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.630736 12939 raft_consensus.cc:385] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.631048 12939 raft_consensus.cc:740] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.630905 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:19.631296 13197 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:19.631847 12895 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.632764 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:19.633580 12893 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:19.633769 12791 leader_election.cc:343] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.639021 12865 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:19.640199 13240 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:19.640797 13240 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:19.641377 13240 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.645664 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:19.648932 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:19.654598 12939 consensus_queue.cc:260] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.656432 12939 ts_tablet_manager.cc:1434] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20260504 20:00:19.656059 13240 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:19.656817 12714 leader_election.cc:343] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.657482 12939 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
W20260504 20:00:19.660329 12791 leader_election.cc:343] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.662057 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:19.662863 12819 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.664435 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:19.659401 13240 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:19.645087 12940 raft_consensus.cc:359] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.669541 13242 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:19.673849 13214 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:19.683717 12940 raft_consensus.cc:385] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.683902 12964 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:19.672802 12819 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:19.684213 12940 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.677029 12893 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:19.680413 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:19.681623 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:19.687175 12745 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.687906 12863 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:19.685360 12940 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.689857 13242 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:19.690460 13242 raft_consensus.cc:697] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:00:19.691174 12791 leader_election.cc:343] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.691754 12791 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:19.692682 13121 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:19.691617 13242 consensus_queue.cc:237] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.694202 12940 ts_tablet_manager.cc:1434] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.080s user 0.005s sys 0.004s
I20260504 20:00:19.694764 12944 ts_tablet_manager.cc:1434] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.082s user 0.000s sys 0.010s
I20260504 20:00:19.696563 12895 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:19.697454 13240 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:19.705683 13240 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.700840 12944 tablet_bootstrap.cc:492] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:19.711556 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:19.714809 13240 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:19.715924 12940 tablet_bootstrap.cc:492] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:19.723362 12716 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:19.724399 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:19.724709 13214 raft_consensus.cc:2804] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:19.725363 13214 raft_consensus.cc:697] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:19.727052 13121 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:19.727761 13121 raft_consensus.cc:697] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:19.726679 13214 consensus_queue.cc:237] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:19.731281 12863 leader_election.cc:343] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.731047 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:19.736800 12865 leader_election.cc:343] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:19.737231 12865 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:19.741328 13240 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:19.731945 13121 consensus_queue.cc:237] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.757855 13216 consensus_queue.cc:1048] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:19.752029 12745 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:19.768810 12939 tablet_bootstrap.cc:654] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.797505 12944 tablet_bootstrap.cc:654] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.805783 12940 tablet_bootstrap.cc:654] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.811826 12645 catalog_manager.cc:5671] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:19.834896 13262 consensus_queue.cc:1048] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260504 20:00:19.838909 12644 catalog_manager.cc:5671] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:19.842165 12939 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:19.842835 12939 ts_tablet_manager.cc:1403] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.186s user 0.013s sys 0.008s
I20260504 20:00:19.848116 13197 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:19.848853 13197 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.853856 13197 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:19.857084 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:19.861865 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:19.862715 12899 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:19.865743 12940 tablet_bootstrap.cc:492] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:19.866479 12940 ts_tablet_manager.cc:1403] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.151s user 0.022s sys 0.008s
I20260504 20:00:19.870820 12716 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:19.872570 13243 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:19.876456 12939 raft_consensus.cc:359] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.880667 12939 raft_consensus.cc:385] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.883533 12939 raft_consensus.cc:740] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.880209 12940 raft_consensus.cc:359] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.884455 12940 raft_consensus.cc:385] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.884768 12940 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.889725 12939 consensus_queue.cc:260] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.892119 12940 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.899503 12939 ts_tablet_manager.cc:1434] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.056s user 0.005s sys 0.003s
I20260504 20:00:19.900712 12939 tablet_bootstrap.cc:492] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:19.904268 12648 catalog_manager.cc:5671] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:19.907362 12940 ts_tablet_manager.cc:1434] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.040s user 0.007s sys 0.001s
I20260504 20:00:19.908716 12940 tablet_bootstrap.cc:492] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:19.924270 13243 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:19.928758 12944 tablet_bootstrap.cc:492] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:19.929613 12944 ts_tablet_manager.cc:1403] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.229s user 0.019s sys 0.018s
I20260504 20:00:19.933578 12940 tablet_bootstrap.cc:654] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.936932 12944 raft_consensus.cc:359] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.937618 12944 raft_consensus.cc:385] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.937901 12944 raft_consensus.cc:740] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.938551 12944 consensus_queue.cc:260] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.946017 12944 ts_tablet_manager.cc:1434] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260504 20:00:19.950171 12939 tablet_bootstrap.cc:654] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.951922 12944 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:19.966413 12944 tablet_bootstrap.cc:654] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:19.968896 12939 tablet_bootstrap.cc:492] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:19.969374 12939 ts_tablet_manager.cc:1403] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.008s
I20260504 20:00:19.974120 12940 tablet_bootstrap.cc:492] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:19.974784 12940 ts_tablet_manager.cc:1403] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.066s user 0.023s sys 0.002s
I20260504 20:00:19.972705 12939 raft_consensus.cc:359] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.978603 12939 raft_consensus.cc:385] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.980746 12939 raft_consensus.cc:740] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.978605 12940 raft_consensus.cc:359] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.981273 12944 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:19.981375 12940 raft_consensus.cc:385] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.981753 12944 ts_tablet_manager.cc:1403] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.005s
I20260504 20:00:19.981454 12939 consensus_queue.cc:260] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.981976 12940 raft_consensus.cc:740] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.982872 12940 consensus_queue.cc:260] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.984799 12939 ts_tablet_manager.cc:1434] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.015s user 0.005s sys 0.005s
I20260504 20:00:19.985570 12944 raft_consensus.cc:359] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.986310 12944 raft_consensus.cc:385] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:19.986568 12944 raft_consensus.cc:740] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:19.987282 12940 ts_tablet_manager.cc:1434] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.012s user 0.005s sys 0.003s
I20260504 20:00:19.987191 12944 consensus_queue.cc:260] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:19.987962 12939 tablet_bootstrap.cc:492] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:19.989557 12944 ts_tablet_manager.cc:1434] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260504 20:00:19.990676 12944 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:19.993552 13214 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:19.998994 12940 tablet_bootstrap.cc:492] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:20.000976 12944 tablet_bootstrap.cc:654] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.001191 13214 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.005373 13214 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.005518 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:20.011271 12894 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:20.006711 12939 tablet_bootstrap.cc:654] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.012703 12716 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:20.008283 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:20.014320 12821 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:20.013622 13197 raft_consensus.cc:2804] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:20.017740 13197 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:20.018268 13197 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.036108 13261 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.036983 13261 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.110169 13121 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.136961 12940 tablet_bootstrap.cc:700] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968: Time spent opening tablet: real 0.113s user 0.000s sys 0.017s
I20260504 20:00:20.111029 13121 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.147465 13261 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.148981 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:20.151468 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:20.152519 12940 tablet_bootstrap.cc:654] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:20.154399 12791 leader_election.cc:343] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.160228 12939 tablet_bootstrap.cc:492] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:20.160833 12939 ts_tablet_manager.cc:1403] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.173s user 0.021s sys 0.009s
I20260504 20:00:20.164173 12939 raft_consensus.cc:359] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.165326 13242 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:20.167105 12939 raft_consensus.cc:385] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.167470 12939 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.170217 13197 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.170173 12939 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.173452 12939 ts_tablet_manager.cc:1434] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.012s user 0.001s sys 0.005s
I20260504 20:00:20.173975 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:20.175432 12821 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.175843 12939 tablet_bootstrap.cc:492] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:20.174676 13197 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.176669 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:20.177734 12893 raft_consensus.cc:2393] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:00:20.186681 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:20.141393 13121 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.194943 12939 tablet_bootstrap.cc:654] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.205703 12824 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.217134 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:20.217934 12895 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.222484 12944 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:20.222990 12944 ts_tablet_manager.cc:1403] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.233s user 0.087s sys 0.092s
I20260504 20:00:20.211921 13262 consensus_queue.cc:1048] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260504 20:00:20.227098 12944 raft_consensus.cc:359] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.228427 12944 raft_consensus.cc:385] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.228698 12944 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.229513 12944 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.235106 13241 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:20.237151 12791 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:00:20.240245 12791 leader_election.cc:343] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.243788 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:20.245270 13262 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:20.246002 13261 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.251133 12944 ts_tablet_manager.cc:1434] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.028s user 0.005s sys 0.005s
I20260504 20:00:20.254014 12895 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:20.250025 13261 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.257302 13261 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.259616 12939 tablet_bootstrap.cc:492] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:20.260113 12939 ts_tablet_manager.cc:1403] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.015s
I20260504 20:00:20.258608 12944 tablet_bootstrap.cc:492] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:20.262006 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:20.264215 12716 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:20.250623 13262 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:20.267885 13213 raft_consensus.cc:2804] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 20:00:20.276429 12791 leader_election.cc:343] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.265012 13241 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260504 20:00:20.278532 13213 raft_consensus.cc:697] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:20.279870 13213 consensus_queue.cc:237] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.282830 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:20.292428 12894 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:20.282171 12821 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:20.294991 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:20.302580 13121 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:20.310669 13121 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:20.311264 13121 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.316708 12940 tablet_bootstrap.cc:492] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:20.317337 12940 ts_tablet_manager.cc:1403] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.319s user 0.024s sys 0.044s
I20260504 20:00:20.348340 12940 raft_consensus.cc:359] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.349426 12940 raft_consensus.cc:385] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.349795 12940 raft_consensus.cc:740] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.350931 12940 consensus_queue.cc:260] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.325939 13240 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260504 20:00:20.366145 12939 raft_consensus.cc:359] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.367961 12939 raft_consensus.cc:385] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.368357 12939 raft_consensus.cc:740] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.370409 12939 consensus_queue.cc:260] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.372017 12944 tablet_bootstrap.cc:654] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.374382 12939 ts_tablet_manager.cc:1434] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.114s user 0.006s sys 0.008s
I20260504 20:00:20.388928 12939 tablet_bootstrap.cc:492] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
W20260504 20:00:20.395506 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 73230.6us
I20260504 20:00:20.410109 12940 ts_tablet_manager.cc:1434] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.092s user 0.009s sys 0.016s
I20260504 20:00:20.439004 12940 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:20.458743 13243 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.472265 12939 tablet_bootstrap.cc:654] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.489161 13121 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.478665 13243 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:20.491487 13288 log.cc:927] Time spent T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260504 20:00:20.499249 13243 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:20.504470 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:20.518471 12644 catalog_manager.cc:5671] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:20.519719 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:20.520663 13121 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:20.521894 12716 leader_election.cc:343] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.525450 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:20.535444 12940 tablet_bootstrap.cc:654] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.545131 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:00:20.549016 12714 leader_election.cc:343] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.549669 12714 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:20.551031 12964 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 20:00:20.552098 12791 leader_election.cc:343] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.552848 13121 consensus_queue.cc:1048] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:20.562456 12944 tablet_bootstrap.cc:492] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:20.562943 12944 ts_tablet_manager.cc:1403] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.305s user 0.011s sys 0.022s
I20260504 20:00:20.565160 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:20.594133 13216 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.596306 13216 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.593039 12895 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.603967 12939 tablet_bootstrap.cc:492] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:20.604590 12939 ts_tablet_manager.cc:1403] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.221s user 0.028s sys 0.015s
I20260504 20:00:20.601400 13241 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.608484 13216 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.627556 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:20.616319 13241 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.629846 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:20.637823 12791 leader_election.cc:343] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:20.640880 12791 leader_election.cc:343] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.641496 12791 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:20.645112 13298 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:20.656533 12940 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:20.656649 12944 raft_consensus.cc:359] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.657326 12940 ts_tablet_manager.cc:1403] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.225s user 0.018s sys 0.030s
I20260504 20:00:20.657634 12944 raft_consensus.cc:385] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.658867 12944 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.660207 12944 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.662814 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:20.664600 12940 raft_consensus.cc:359] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.665267 12940 raft_consensus.cc:385] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.663503 12944 ts_tablet_manager.cc:1434] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.100s user 0.006s sys 0.009s
I20260504 20:00:20.665621 12940 raft_consensus.cc:740] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.666528 12940 consensus_queue.cc:260] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:20.667240 12714 leader_election.cc:343] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:20.668268 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 49769.2us
I20260504 20:00:20.666739 12944 tablet_bootstrap.cc:492] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:20.670485 12940 ts_tablet_manager.cc:1434] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260504 20:00:20.671615 12940 tablet_bootstrap.cc:492] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:20.685014 12939 raft_consensus.cc:359] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.694870 12939 raft_consensus.cc:385] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.702600 12939 raft_consensus.cc:740] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.701342 12895 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:20.705189 12939 consensus_queue.cc:260] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.710934 12939 ts_tablet_manager.cc:1434] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.106s user 0.007s sys 0.003s
I20260504 20:00:20.712152 12939 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:20.710798 13241 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.717898 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:20.718858 12898 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:20.723505 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:20.725787 13298 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:20.726482 13298 raft_consensus.cc:697] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:20.727735 13298 consensus_queue.cc:237] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.751729 12939 tablet_bootstrap.cc:654] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.761099 12939 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:20.763890 12939 ts_tablet_manager.cc:1403] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.011s
I20260504 20:00:20.764693 13309 consensus_queue.cc:1048] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:20.765583 12944 tablet_bootstrap.cc:654] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.770738 12716 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:20.776098 13241 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:20.776568 13241 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:20.777048 13241 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.777104 12940 tablet_bootstrap.cc:654] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.810058 13294 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.811599 12939 raft_consensus.cc:359] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.812734 12939 raft_consensus.cc:385] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.813150 12939 raft_consensus.cc:740] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.814823 12939 consensus_queue.cc:260] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.810674 13294 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.817662 13213 consensus_queue.cc:1048] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260504 20:00:20.820822 12939 ts_tablet_manager.cc:1434] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.057s user 0.006s sys 0.006s
I20260504 20:00:20.821713 13294 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.833395 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:20.841500 13241 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.848709 13242 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.847184 12939 tablet_bootstrap.cc:492] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:20.849454 13242 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.852044 13242 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:20.858435 12939 tablet_bootstrap.cc:654] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:20.862190 12791 leader_election.cc:343] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.862079 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:20.877708 12940 tablet_bootstrap.cc:492] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:20.878597 12940 ts_tablet_manager.cc:1403] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.207s user 0.018s sys 0.008s
I20260504 20:00:20.879395 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:20.879060 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:00:20.887775 12863 leader_election.cc:343] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:20.890720 12714 leader_election.cc:343] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.893067 13241 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:20.895714 12791 leader_election.cc:343] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.896262 12791 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:20.896709 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:20.899186 13262 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:20.924181 12648 catalog_manager.cc:5671] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:20.928045 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:20.928969 12895 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.934347 12944 tablet_bootstrap.cc:492] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:20.934880 12944 ts_tablet_manager.cc:1403] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.268s user 0.088s sys 0.083s
I20260504 20:00:20.936477 13216 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:20.937063 13216 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.939424 13216 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.940311 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:20.941177 12748 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:20.941761 12939 tablet_bootstrap.cc:492] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:20.942288 12939 ts_tablet_manager.cc:1403] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.095s user 0.034s sys 0.028s
I20260504 20:00:20.942719 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:20.943740 13216 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:20.944226 13216 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:20.944691 13216 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.949679 12940 raft_consensus.cc:359] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.950933 12939 raft_consensus.cc:359] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.972012 12939 raft_consensus.cc:385] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.960918 13216 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.972361 12939 raft_consensus.cc:740] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.973157 12939 consensus_queue.cc:260] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.950992 12944 raft_consensus.cc:359] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.974388 13216 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:20.974833 12944 raft_consensus.cc:385] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.975154 12944 raft_consensus.cc:740] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.975806 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:20.975898 12944 consensus_queue.cc:260] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.976641 12748 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:20.977923 12940 raft_consensus.cc:385] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:20.978235 12940 raft_consensus.cc:740] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:20.978847 12940 consensus_queue.cc:260] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.981065 12940 ts_tablet_manager.cc:1434] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.102s user 0.005s sys 0.000s
I20260504 20:00:20.982105 12940 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:20.984681 12939 ts_tablet_manager.cc:1434] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.042s user 0.005s sys 0.011s
I20260504 20:00:20.987793 12944 ts_tablet_manager.cc:1434] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.053s user 0.017s sys 0.011s
I20260504 20:00:20.989171 12944 tablet_bootstrap.cc:492] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:20.990003 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:20.991454 12865 leader_election.cc:343] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:20.995841 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:20.994405 12895 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:20.997255 12944 tablet_bootstrap.cc:654] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.000031 13293 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:20.994436 12940 tablet_bootstrap.cc:654] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:20.998951 12939 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
W20260504 20:00:21.016263 12791 leader_election.cc:343] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.018059 12716 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:21.020083 13241 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:21.021232 13241 raft_consensus.cc:697] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:21.023154 12944 tablet_bootstrap.cc:492] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:21.023605 12944 ts_tablet_manager.cc:1403] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.021s
I20260504 20:00:21.023358 13241 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:20.998837 12748 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:21.013821 13242 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.025720 12939 tablet_bootstrap.cc:654] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.049855 12940 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:21.026937 12944 raft_consensus.cc:359] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.050660 12940 ts_tablet_manager.cc:1403] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.069s user 0.015s sys 0.005s
I20260504 20:00:21.050876 12944 raft_consensus.cc:385] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.051157 12944 raft_consensus.cc:740] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.051946 13242 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.051949 12944 consensus_queue.cc:260] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.053521 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.037290 13309 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.055020 13242 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:21.055262 12944 ts_tablet_manager.cc:1434] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.031s user 0.014s sys 0.011s
I20260504 20:00:21.055845 13262 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:21.055603 13309 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.056553 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:21.059830 12863 leader_election.cc:343] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.056574 13262 raft_consensus.cc:697] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:21.068199 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:21.080600 12944 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:21.085211 12940 raft_consensus.cc:359] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.085815 12940 raft_consensus.cc:385] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.086107 12940 raft_consensus.cc:740] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.086700 12940 consensus_queue.cc:260] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.088280 12940 ts_tablet_manager.cc:1434] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.037s user 0.005s sys 0.008s
I20260504 20:00:21.096513 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:00:21.099575 12791 leader_election.cc:343] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.061674 13262 consensus_queue.cc:237] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.108068 12940 tablet_bootstrap.cc:492] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:21.095589 12824 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:21.122282 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:21.123152 12899 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:21.124521 12716 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.120911 13309 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:21.137324 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:21.137696 12939 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:21.138386 12939 ts_tablet_manager.cc:1403] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.140s user 0.005s sys 0.024s
I20260504 20:00:21.128275 13241 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:21.147532 12646 catalog_manager.cc:5671] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:00:21.150650 12865 leader_election.cc:343] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.151155 12865 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:21.156066 13213 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.156762 13242 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:21.152029 13241 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:21.162580 13241 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.158967 13213 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.176390 13213 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:21.181103 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:21.180095 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:21.185626 12716 leader_election.cc:343] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
W20260504 20:00:21.186276 12714 leader_election.cc:343] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.186772 12714 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:21.188019 13323 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:21.196368 12939 raft_consensus.cc:359] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.197232 12939 raft_consensus.cc:385] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.197516 12939 raft_consensus.cc:740] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.200137 12939 consensus_queue.cc:260] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.202389 12939 ts_tablet_manager.cc:1434] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.064s user 0.009s sys 0.002s
I20260504 20:00:21.203635 12939 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:21.209939 12944 tablet_bootstrap.cc:654] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.218681 12940 tablet_bootstrap.cc:654] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.222457 12646 catalog_manager.cc:5671] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:21.227036 13241 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.231671 13241 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:21.232673 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:21.233242 12899 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.234537 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:21.235288 12825 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.243731 12899 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:21.248221 12825 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
W20260504 20:00:21.250981 12791 consensus_peers.cc:597] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Couldn't send request to peer 82787d8e64d24ca885ca517aa8ed1be2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:21.250023 12716 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.252425 13214 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:21.253414 13214 raft_consensus.cc:697] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:21.249192 13241 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.254904 13214 consensus_queue.cc:237] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.256103 13241 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.260210 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:21.260349 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:21.261256 12825 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
W20260504 20:00:21.262187 12716 leader_election.cc:343] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.262869 12940 tablet_bootstrap.cc:492] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:21.263195 12714 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:21.263417 12940 ts_tablet_manager.cc:1403] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.156s user 0.018s sys 0.011s
I20260504 20:00:21.264173 13309 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:21.264671 13309 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:21.265269 12939 tablet_bootstrap.cc:654] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.265147 13309 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.269569 12940 raft_consensus.cc:359] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.271147 12940 raft_consensus.cc:385] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.271431 12940 raft_consensus.cc:740] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.272512 12944 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:21.272109 12940 consensus_queue.cc:260] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.273005 12944 ts_tablet_manager.cc:1403] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.193s user 0.013s sys 0.005s
I20260504 20:00:21.284828 12944 raft_consensus.cc:359] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.290589 12940 ts_tablet_manager.cc:1434] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.027s user 0.011s sys 0.000s
I20260504 20:00:21.293215 12940 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:21.299157 13262 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:21.277992 13241 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:21.302387 12940 tablet_bootstrap.cc:654] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.285712 12944 raft_consensus.cc:385] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.303413 12944 raft_consensus.cc:740] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.304049 12944 consensus_queue.cc:260] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.298419 13309 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.309319 13309 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:21.310201 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:21.310717 12825 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.315186 12646 catalog_manager.cc:5671] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:21.319321 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:21.319763 12899 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.318768 12825 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:21.322975 12714 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.325497 12939 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:21.326046 12939 ts_tablet_manager.cc:1403] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.123s user 0.021s sys 0.025s
I20260504 20:00:21.329553 13214 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:21.330156 13214 raft_consensus.cc:697] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:21.331064 13214 consensus_queue.cc:237] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.336455 12944 ts_tablet_manager.cc:1434] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20260504 20:00:21.341657 12939 raft_consensus.cc:359] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.342471 12939 raft_consensus.cc:385] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.342736 12939 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.343535 12944 tablet_bootstrap.cc:492] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:21.345126 12939 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.363408 13293 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.364198 13293 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.368352 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:21.369263 12749 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:21.372217 12865 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.373136 13242 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:21.373728 13293 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:21.373528 13242 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:21.375113 13242 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.377230 12899 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:21.399922 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:21.400840 12819 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:21.401779 12940 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:21.404098 12940 ts_tablet_manager.cc:1403] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.007s
I20260504 20:00:21.417042 12944 tablet_bootstrap.cc:654] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.428282 13242 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.430544 12940 raft_consensus.cc:359] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.431283 12940 raft_consensus.cc:385] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.431613 12940 raft_consensus.cc:740] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.437989 13242 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:21.442262 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:21.443084 12749 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.453495 12749 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:21.454813 12865 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.456290 13242 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:21.457485 13242 raft_consensus.cc:697] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:21.458945 13242 consensus_queue.cc:237] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.469902 12940 consensus_queue.cc:260] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.474025 12940 ts_tablet_manager.cc:1434] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.070s user 0.008s sys 0.002s
I20260504 20:00:21.475128 12940 tablet_bootstrap.cc:492] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:21.480307 13242 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.481170 13242 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.472214 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:21.491827 12939 ts_tablet_manager.cc:1434] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.165s user 0.037s sys 0.031s
I20260504 20:00:21.492345 12821 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.493633 12939 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:21.500476 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:21.500972 12940 tablet_bootstrap.cc:654] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:21.505149 12865 leader_election.cc:343] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.505642 13242 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:21.507957 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:21.508618 12825 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:21.514065 12863 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:21.518157 13293 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:21.520136 13293 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:21.520668 13293 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.524468 12646 catalog_manager.cc:5671] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:21.531703 12894 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.538726 13293 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.540408 12821 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:21.541327 12944 tablet_bootstrap.cc:492] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:21.541839 12944 ts_tablet_manager.cc:1403] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.204s user 0.006s sys 0.026s
I20260504 20:00:21.545131 12824 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.544911 12645 catalog_manager.cc:5671] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:21.550114 12939 tablet_bootstrap.cc:654] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.553297 12940 tablet_bootstrap.cc:492] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:21.553858 12940 ts_tablet_manager.cc:1403] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.012s
I20260504 20:00:21.554257 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:00:21.555450 12865 leader_election.cc:343] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.562180 12944 raft_consensus.cc:359] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.562939 12944 raft_consensus.cc:385] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.563271 12944 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.563910 12944 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.556144 13293 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:21.566522 13262 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:21.568593 12940 raft_consensus.cc:359] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.576140 12944 ts_tablet_manager.cc:1434] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260504 20:00:21.569324 12940 raft_consensus.cc:385] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.576601 12940 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.577307 12940 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.579437 12940 ts_tablet_manager.cc:1434] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260504 20:00:21.580880 12944 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:21.583168 13241 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:21.586706 13290 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.587229 13290 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.589376 13290 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:21.593288 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:21.602811 13241 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:21.603327 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:21.604230 12749 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:21.604646 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:21.605980 12940 tablet_bootstrap.cc:492] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:21.608791 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.610752 13242 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:21.611156 13242 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:21.630515 12939 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:21.631273 12939 ts_tablet_manager.cc:1403] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.138s user 0.015s sys 0.021s
I20260504 20:00:21.610085 12821 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.619448 13242 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.635390 12944 tablet_bootstrap.cc:654] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.640583 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:21.652837 12863 leader_election.cc:343] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.658551 12940 tablet_bootstrap.cc:654] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.685027 13216 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260504 20:00:21.702822 12939 raft_consensus.cc:359] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.703949 12939 raft_consensus.cc:385] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.704325 12939 raft_consensus.cc:740] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.696987 13242 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.712827 12939 consensus_queue.cc:260] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.711611 12821 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:21.724705 13242 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:21.729329 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:21.730021 12749 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.731173 12944 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:21.731792 12944 ts_tablet_manager.cc:1403] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.151s user 0.018s sys 0.009s
I20260504 20:00:21.742689 12939 ts_tablet_manager.cc:1434] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.111s user 0.009s sys 0.000s
I20260504 20:00:21.743742 12939 tablet_bootstrap.cc:492] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:21.737089 12863 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:21.746215 13242 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:21.746415 12749 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:21.746737 13242 raft_consensus.cc:697] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:21.747678 13242 consensus_queue.cc:237] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.748064 12944 raft_consensus.cc:359] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.748726 12944 raft_consensus.cc:385] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.748988 12944 raft_consensus.cc:740] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.749583 12944 consensus_queue.cc:260] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.751677 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.754480 13290 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:21.758531 13290 raft_consensus.cc:697] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:21.759423 13290 consensus_queue.cc:237] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.761495 13323 consensus_queue.cc:1048] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:21.762390 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:21.766423 12944 ts_tablet_manager.cc:1434] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.034s user 0.009s sys 0.000s
W20260504 20:00:21.767961 12863 leader_election.cc:343] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:21.776924 12944 tablet_bootstrap.cc:492] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:21.799130 12940 tablet_bootstrap.cc:492] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:21.799678 12940 ts_tablet_manager.cc:1403] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.198s user 0.017s sys 0.006s
I20260504 20:00:21.801942 13197 consensus_queue.cc:1048] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:21.798732 13214 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.805016 13214 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.807996 13214 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:21.812151 13213 consensus_queue.cc:1048] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:21.830199 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:21.858444 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:21.864988 12939 tablet_bootstrap.cc:654] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.876209 12714 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:21.885855 12940 raft_consensus.cc:359] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.886636 12940 raft_consensus.cc:385] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:21.886970 12940 raft_consensus.cc:740] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:21.888700 12944 tablet_bootstrap.cc:654] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.887558 12940 consensus_queue.cc:260] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.893431 12645 catalog_manager.cc:5671] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:21.898353 12940 ts_tablet_manager.cc:1434] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.098s user 0.013s sys 0.004s
I20260504 20:00:21.899539 12940 tablet_bootstrap.cc:492] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:21.891633 13323 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:21.904399 13352 consensus_queue.cc:1048] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260504 20:00:21.932919 12748 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.943970 12940 tablet_bootstrap.cc:654] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:21.953940 12939 tablet_bootstrap.cc:492] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:21.954497 12939 ts_tablet_manager.cc:1403] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.211s user 0.017s sys 0.035s
I20260504 20:00:21.967823 12646 catalog_manager.cc:5671] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:21.988436 13262 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:21.989248 13262 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.993626 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:21.994396 12749 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:21.994498 13262 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:21.996071 12791 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:21.998041 13347 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:21.998638 13347 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:21.994194 12939 raft_consensus.cc:359] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:21.999099 13347 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:21.999570 12939 raft_consensus.cc:385] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.000864 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:21.999905 12939 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.003082 12939 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.008057 12939 ts_tablet_manager.cc:1434] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.053s user 0.012s sys 0.000s
I20260504 20:00:22.011073 12939 tablet_bootstrap.cc:492] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:22.001865 12894 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:22.079581 13294 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.080257 13294 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.091711 13294 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:22.124891 12944 tablet_bootstrap.cc:492] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:22.125213 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:22.125444 12944 ts_tablet_manager.cc:1403] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.349s user 0.027s sys 0.009s
I20260504 20:00:22.126312 12940 tablet_bootstrap.cc:492] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:22.126776 12940 ts_tablet_manager.cc:1403] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.227s user 0.063s sys 0.052s
I20260504 20:00:22.131212 12939 tablet_bootstrap.cc:654] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.143524 12940 raft_consensus.cc:359] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.144308 12940 raft_consensus.cc:385] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.144593 12940 raft_consensus.cc:740] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.145303 12940 consensus_queue.cc:260] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.146970 13347 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.149513 13347 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:22.154222 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:22.155710 12748 raft_consensus.cc:2393] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:22.160468 12940 ts_tablet_manager.cc:1434] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20260504 20:00:22.160368 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:22.161429 12896 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.168497 13242 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:22.170356 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:22.171336 12746 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.178709 12940 tablet_bootstrap.cc:492] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:22.184563 13339 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:22.204574 12791 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:22.209544 13294 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:22.242076 12865 consensus_peers.cc:597] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 -> Peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Couldn't send request to peer 82787d8e64d24ca885ca517aa8ed1be2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:22.243027 12944 raft_consensus.cc:359] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.244992 12944 raft_consensus.cc:385] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.246029 12944 raft_consensus.cc:740] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.257844 12944 consensus_queue.cc:260] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.266163 13242 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.266830 13294 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:22.266769 13242 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.268867 13242 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:22.269481 12896 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:22.270393 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:22.270882 12791 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:22.272040 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:22.273270 12865 leader_election.cc:343] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.259717 12746 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:22.272766 13294 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:22.274678 12944 ts_tablet_manager.cc:1434] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.149s user 0.009s sys 0.000s
W20260504 20:00:22.275799 12863 leader_election.cc:343] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.276232 12863 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:22.276682 12940 tablet_bootstrap.cc:654] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.277489 13293 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:22.274237 13294 raft_consensus.cc:697] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:22.279703 13294 consensus_queue.cc:237] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.289002 12939 tablet_bootstrap.cc:492] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:22.289577 12939 ts_tablet_manager.cc:1403] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.281s user 0.042s sys 0.066s
I20260504 20:00:22.290903 12944 tablet_bootstrap.cc:492] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:22.305552 13295 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:22.308354 12939 raft_consensus.cc:359] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.309147 12939 raft_consensus.cc:385] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.309454 12939 raft_consensus.cc:740] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.310963 12939 consensus_queue.cc:260] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.317102 12944 tablet_bootstrap.cc:654] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.328526 12939 ts_tablet_manager.cc:1434] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.039s user 0.005s sys 0.001s
I20260504 20:00:22.331318 12939 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:22.394037 13361 consensus_queue.cc:1048] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:22.395939 13361 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:22.407294 12939 tablet_bootstrap.cc:654] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.408269 12940 tablet_bootstrap.cc:492] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:22.408854 12940 ts_tablet_manager.cc:1403] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.247s user 0.009s sys 0.024s
I20260504 20:00:22.413587 13241 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.414505 12944 tablet_bootstrap.cc:492] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:22.415095 12944 ts_tablet_manager.cc:1403] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.128s user 0.010s sys 0.023s
I20260504 20:00:22.414525 13241 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.417680 12644 catalog_manager.cc:5671] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:22.431289 13241 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:22.432988 12940 raft_consensus.cc:359] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.435199 12944 raft_consensus.cc:359] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.435350 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:22.438462 12944 raft_consensus.cc:385] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.438788 12944 raft_consensus.cc:740] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.439541 12944 consensus_queue.cc:260] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.438076 12940 raft_consensus.cc:385] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.440575 12940 raft_consensus.cc:740] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.441543 12940 consensus_queue.cc:260] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.442715 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:22.446295 12714 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:22.447186 12944 ts_tablet_manager.cc:1434] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.032s user 0.007s sys 0.002s
I20260504 20:00:22.447075 13352 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:22.444123 12940 ts_tablet_manager.cc:1434] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260504 20:00:22.458046 13290 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.458776 13290 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:22.461483 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 58810.7us
I20260504 20:00:22.461987 13290 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:22.463056 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:22.464464 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:22.465342 12821 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:22.466449 12944 tablet_bootstrap.cc:492] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
W20260504 20:00:22.467072 12865 leader_election.cc:343] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.469585 12863 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:22.465581 12940 tablet_bootstrap.cc:492] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:22.482183 13295 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:22.482769 13295 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:22.483281 13295 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.494307 12944 tablet_bootstrap.cc:654] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.504810 12939 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:22.506412 12939 ts_tablet_manager.cc:1403] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.175s user 0.015s sys 0.027s
I20260504 20:00:22.517306 12939 raft_consensus.cc:359] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.518523 12939 raft_consensus.cc:385] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.518834 12939 raft_consensus.cc:740] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.519529 12939 consensus_queue.cc:260] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.521391 13295 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.527448 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:22.528038 12824 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.528184 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:22.533674 12939 ts_tablet_manager.cc:1434] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.027s user 0.004s sys 0.003s
W20260504 20:00:22.534579 12865 leader_election.cc:343] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.536161 12939 tablet_bootstrap.cc:492] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:22.545974 12824 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:22.550202 12863 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:22.551455 13361 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:22.554203 13361 raft_consensus.cc:697] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:22.560726 12940 tablet_bootstrap.cc:654] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.529515 13295 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:22.562678 13361 consensus_queue.cc:237] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.568626 12944 tablet_bootstrap.cc:492] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:22.572422 12944 ts_tablet_manager.cc:1403] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.039s
I20260504 20:00:22.574713 12939 tablet_bootstrap.cc:654] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.577018 12944 raft_consensus.cc:359] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.580574 12944 raft_consensus.cc:385] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.581079 12940 tablet_bootstrap.cc:492] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:22.581451 12944 raft_consensus.cc:740] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.581619 12940 ts_tablet_manager.cc:1403] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.116s user 0.008s sys 0.010s
I20260504 20:00:22.586011 12940 raft_consensus.cc:359] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.586794 12940 raft_consensus.cc:385] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.587182 12940 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.587996 12940 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.590010 12940 ts_tablet_manager.cc:1434] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260504 20:00:22.590808 12944 consensus_queue.cc:260] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.604667 12944 ts_tablet_manager.cc:1434] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.031s user 0.001s sys 0.005s
I20260504 20:00:22.606002 12944 tablet_bootstrap.cc:492] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:22.614471 12746 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.603289 12940 tablet_bootstrap.cc:492] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:22.631044 12940 tablet_bootstrap.cc:654] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.634145 13352 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.634912 13352 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.635975 12644 catalog_manager.cc:5671] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:22.643836 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:22.643234 13352 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:22.646073 12714 leader_election.cc:343] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.648859 12944 tablet_bootstrap.cc:654] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.649504 12939 tablet_bootstrap.cc:492] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:22.650075 12939 ts_tablet_manager.cc:1403] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.114s user 0.019s sys 0.018s
I20260504 20:00:22.655079 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:22.657008 12716 leader_election.cc:343] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.657582 12716 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:22.659004 13241 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:22.664451 12940 tablet_bootstrap.cc:492] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
W20260504 20:00:22.665484 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:22.665779 12940 ts_tablet_manager.cc:1403] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.073s user 0.020s sys 0.001s
I20260504 20:00:22.667632 12939 raft_consensus.cc:359] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.668798 12939 raft_consensus.cc:385] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.669098 12939 raft_consensus.cc:740] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.669737 12939 consensus_queue.cc:260] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.671283 12944 tablet_bootstrap.cc:492] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:22.671710 12944 ts_tablet_manager.cc:1403] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.066s user 0.014s sys 0.006s
I20260504 20:00:22.672199 12939 ts_tablet_manager.cc:1434] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20260504 20:00:22.675212 12944 raft_consensus.cc:359] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.675894 12944 raft_consensus.cc:385] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.676239 12939 tablet_bootstrap.cc:492] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:22.679821 12940 raft_consensus.cc:359] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.681335 12940 raft_consensus.cc:385] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.681679 12940 raft_consensus.cc:740] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.682648 12940 consensus_queue.cc:260] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.685050 12940 ts_tablet_manager.cc:1434] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260504 20:00:22.676211 12944 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.686017 12940 tablet_bootstrap.cc:492] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:22.686787 12944 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.687575 12939 tablet_bootstrap.cc:654] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.690724 12944 ts_tablet_manager.cc:1434] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.019s user 0.009s sys 0.001s
I20260504 20:00:22.696918 12940 tablet_bootstrap.cc:654] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.699838 12944 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:22.702731 12939 tablet_bootstrap.cc:492] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:22.703227 12939 ts_tablet_manager.cc:1403] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.013s
I20260504 20:00:22.708544 13346 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.709177 13346 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.712602 13346 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:22.719714 13346 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.721128 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:22.720538 13346 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.725082 13361 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.725742 13361 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.726531 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:22.729734 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:22.730335 12749 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:22.730907 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:22.731545 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:22.731734 12824 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:22.732372 13339 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:22.732884 13339 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:22.733314 13339 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.733923 13361 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:22.741878 12944 tablet_bootstrap.cc:654] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
W20260504 20:00:22.762323 12791 leader_election.cc:343] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.764135 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:22.764117 13346 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:22.719745 12939 raft_consensus.cc:359] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.769744 12939 raft_consensus.cc:385] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.770102 12939 raft_consensus.cc:740] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.770752 12939 consensus_queue.cc:260] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.772308 12939 ts_tablet_manager.cc:1434] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.069s user 0.003s sys 0.006s
I20260504 20:00:22.772657 12944 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:22.773414 12939 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
W20260504 20:00:22.774066 12791 leader_election.cc:343] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.774606 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:22.775413 13347 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 20:00:22.775843 12791 leader_election.cc:343] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.773453 12944 ts_tablet_manager.cc:1403] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.074s user 0.013s sys 0.009s
I20260504 20:00:22.787134 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:22.756624 13339 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.791854 12896 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:22.798425 12944 raft_consensus.cc:359] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.799443 12944 raft_consensus.cc:385] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.799873 12944 raft_consensus.cc:740] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.802677 12944 consensus_queue.cc:260] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.806056 12944 ts_tablet_manager.cc:1434] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260504 20:00:22.806639 12939 tablet_bootstrap.cc:654] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.807614 12944 tablet_bootstrap.cc:492] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:22.810348 12791 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:22.810470 13323 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.821969 13347 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:22.822460 13347 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:22.822072 13323 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.822844 13347 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.824361 13323 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:22.814733 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:22.825901 12747 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.826215 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:22.827070 12824 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:22.829249 12714 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:22.832593 13241 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:22.833287 13241 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:22.833765 13241 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.834355 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:22.834933 12823 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.842113 13331 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:22.845422 12823 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:22.846961 12863 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:22.832990 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:22.848169 13361 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:22.848308 12896 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:22.848845 13361 raft_consensus.cc:697] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:22.849947 13361 consensus_queue.cc:237] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.854491 12747 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:22.867647 13346 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:22.892951 13241 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.832597 13347 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.903824 13339 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:22.905886 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:00:22.907683 12791 leader_election.cc:343] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.911594 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:22.912468 12896 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.914485 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:22.915252 12824 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:22.923875 13293 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.924453 13293 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.926452 13347 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:22.928279 13293 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:22.929001 13241 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:22.928272 12944 tablet_bootstrap.cc:654] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.929517 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:22.930886 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:22.931479 12894 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
W20260504 20:00:22.931628 12863 leader_election.cc:343] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.924733 12896 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:22.933234 12716 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:22.934204 13241 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:22.934840 13241 raft_consensus.cc:697] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:22.935709 13241 consensus_queue.cc:237] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.937083 12940 tablet_bootstrap.cc:492] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:22.937569 12940 ts_tablet_manager.cc:1403] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.252s user 0.070s sys 0.098s
I20260504 20:00:22.938194 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:22.940063 12894 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
W20260504 20:00:22.940529 12865 leader_election.cc:343] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:22.941031 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:22.942431 12940 raft_consensus.cc:359] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.942690 12791 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:22.943768 13369 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:22.944389 13369 raft_consensus.cc:697] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:22.946619 12940 raft_consensus.cc:385] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.946950 12940 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.947635 12940 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.948575 13361 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:22.945612 13369 consensus_queue.cc:237] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.950564 12940 ts_tablet_manager.cc:1434] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260504 20:00:22.951745 12940 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:22.965456 12944 tablet_bootstrap.cc:492] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:22.965847 12940 tablet_bootstrap.cc:654] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:22.966053 12944 ts_tablet_manager.cc:1403] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.159s user 0.014s sys 0.002s
I20260504 20:00:22.978832 12824 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:22.977317 12646 catalog_manager.cc:5671] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:22.990952 13339 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:22.993664 13339 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.995811 12944 raft_consensus.cc:359] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:22.996596 12944 raft_consensus.cc:385] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:22.996870 12944 raft_consensus.cc:740] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:22.999562 13339 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:22.997558 12944 consensus_queue.cc:260] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.004281 12944 ts_tablet_manager.cc:1434] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260504 20:00:23.005026 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:23.005862 12747 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:23.007369 12944 tablet_bootstrap.cc:492] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:23.008139 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:23.009909 12863 leader_election.cc:343] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:23.017289 12865 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:23.018476 13339 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:23.018972 13339 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:23.019532 13339 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.025077 12940 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:23.025552 12940 ts_tablet_manager.cc:1403] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.074s user 0.016s sys 0.012s
I20260504 20:00:23.028360 12939 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:23.028936 12939 ts_tablet_manager.cc:1403] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.256s user 0.036s sys 0.067s
I20260504 20:00:23.031270 12940 raft_consensus.cc:359] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.032081 12940 raft_consensus.cc:385] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:23.032394 12940 raft_consensus.cc:740] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
I20260504 20:00:23.032629 12939 raft_consensus.cc:359] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.033614 12939 raft_consensus.cc:385] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:23.033964 12939 raft_consensus.cc:740] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:23.034364 13339 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.034688 12939 consensus_queue.cc:260] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.037204 12939 ts_tablet_manager.cc:1434] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260504 20:00:23.038192 12944 tablet_bootstrap.cc:654] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:23.038540 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:23.038717 13339 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:23.039273 12747 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.044936 12940 consensus_queue.cc:260] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.046324 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:23.047668 13361 consensus_queue.cc:1048] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 20:00:23.052693 12863 leader_election.cc:343] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:23.053534 12939 tablet_bootstrap.cc:492] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:23.061244 12940 ts_tablet_manager.cc:1434] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.035s user 0.007s sys 0.003s
I20260504 20:00:23.057271 12747 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:23.063266 12940 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968: Bootstrap starting.
I20260504 20:00:23.064081 12865 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:23.065191 13339 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:23.066063 13339 raft_consensus.cc:697] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:23.067217 13339 consensus_queue.cc:237] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.068104 12939 tablet_bootstrap.cc:654] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:23.077229 12747 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.093732 12944 tablet_bootstrap.cc:492] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:23.094738 12944 ts_tablet_manager.cc:1403] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.088s user 0.031s sys 0.022s
I20260504 20:00:23.106091 12644 catalog_manager.cc:5671] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:23.115146 12940 tablet_bootstrap.cc:654] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:23.117929 12944 raft_consensus.cc:359] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.119062 12944 raft_consensus.cc:385] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:23.102421 13339 consensus_queue.cc:1048] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260504 20:00:23.119428 12944 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:23.120428 12944 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.123178 12944 ts_tablet_manager.cc:1434] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.028s user 0.007s sys 0.001s
I20260504 20:00:23.124604 12944 tablet_bootstrap.cc:492] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:23.123191 12939 tablet_bootstrap.cc:492] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:23.126124 12939 ts_tablet_manager.cc:1403] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.088s user 0.019s sys 0.008s
I20260504 20:00:23.126327 13323 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:23.127012 13323 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.130446 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:23.131803 12714 leader_election.cc:343] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Illegal state: must be running to vote when last-logged opid is not known
I20260504 20:00:23.135281 12939 raft_consensus.cc:359] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.136327 12939 raft_consensus.cc:385] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:23.136682 12939 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:23.137490 12939 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.142498 13352 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:23.143226 13352 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.145287 12939 ts_tablet_manager.cc:1434] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20260504 20:00:23.159198 12645 catalog_manager.cc:5671] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:23.179296 12944 tablet_bootstrap.cc:654] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:23.179584 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:23.182030 12939 tablet_bootstrap.cc:492] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Bootstrap starting.
I20260504 20:00:23.155351 13352 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:23.183835 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:23.200991 13242 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:23.259341 13242 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.268262 13369 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:23.269002 13369 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.200412 13351 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:23.208102 13323 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:23.214622 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:23.308580 12893 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:23.335203 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:23.345733 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:23.346692 12825 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:23.347506 12716 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:23.349699 12940 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968: No bootstrap required, opened a new log
I20260504 20:00:23.350931 12863 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:23.316926 13242 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:23.359273 12940 ts_tablet_manager.cc:1403] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968: Time spent bootstrapping tablet: real 0.296s user 0.021s sys 0.041s
I20260504 20:00:23.303203 13351 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.361056 12750 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:23.362489 13351 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:23.368423 13295 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:23.387228 13197 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:23.390697 13197 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:23.393565 12646 catalog_manager.cc:5671] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:23.398619 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:23.401299 12714 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:23.390029 13295 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:23.404219 13295 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.405596 12939 tablet_bootstrap.cc:654] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:23.386436 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:23.391180 13197 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.401086 13369 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:23.409159 13398 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:23.424618 13369 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:00:23.432466 13395 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.104s user 0.002s sys 0.000s
W20260504 20:00:23.451190 13400 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260504 20:00:23.452551 12944 tablet_bootstrap.cc:492] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:23.453117 12944 ts_tablet_manager.cc:1403] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.329s user 0.020s sys 0.018s
I20260504 20:00:23.463696 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:23.467736 12821 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 0.
I20260504 20:00:23.471963 12944 raft_consensus.cc:359] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.472985 12944 raft_consensus.cc:385] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:23.475114 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:23.476204 12747 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:23.481419 12944 raft_consensus.cc:740] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:23.441279 13369 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.499840 12939 tablet_bootstrap.cc:492] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: No bootstrap required, opened a new log
I20260504 20:00:23.500568 12939 ts_tablet_manager.cc:1403] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent bootstrapping tablet: real 0.326s user 0.024s sys 0.011s
I20260504 20:00:23.502045 12944 consensus_queue.cc:260] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.512400 12944 ts_tablet_manager.cc:1434] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.059s user 0.010s sys 0.000s
I20260504 20:00:23.503670 13290 consensus_queue.cc:1048] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260504 20:00:23.516736 13369 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:23.543689 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 80574.4us
I20260504 20:00:23.542305 12940 raft_consensus.cc:359] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.549067 12940 raft_consensus.cc:385] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:23.545060 13323 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:23.549446 12940 raft_consensus.cc:740] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Initialized, Role: FOLLOWER
W20260504 20:00:23.560315 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 89409.3us
I20260504 20:00:23.564915 13410 consensus_queue.cc:1048] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260504 20:00:23.565404 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:23.566550 12748 raft_consensus.cc:2393] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:23.567760 12714 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:23.570111 13351 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:23.570708 13351 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:23.571262 13351 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.571010 13323 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.573766 13323 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:23.558557 12940 consensus_queue.cc:260] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.596690 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:23.597967 12893 raft_consensus.cc:2393] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:23.550375 13197 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.616060 12940 ts_tablet_manager.cc:1434] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968: Time spent starting tablet: real 0.256s user 0.010s sys 0.008s
I20260504 20:00:23.642730 13295 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.645118 12898 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:23.647109 13197 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:23.647621 13295 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:23.649008 12791 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:23.688959 13418 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:23.689785 12939 raft_consensus.cc:359] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:23.692178 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 87744.9us
I20260504 20:00:23.690601 12939 raft_consensus.cc:385] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:23.692951 12939 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Initialized, Role: FOLLOWER
I20260504 20:00:23.679519 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:23.695659 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:23.689572 13418 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:23.697274 12939 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.686898 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:00:23.710108 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 351384us
I20260504 20:00:23.702736 13418 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.717381 12939 ts_tablet_manager.cc:1434] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent starting tablet: real 0.216s user 0.010s sys 0.001s
I20260504 20:00:23.714792 12819 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.721593 13417 consensus_queue.cc:1048] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260504 20:00:23.827345 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:23.834136 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:23.835326 12895 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.833210 13409 consensus_queue.cc:1048] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
W20260504 20:00:23.842643 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 145780us
I20260504 20:00:23.848237 13351 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.851356 13351 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:23.853600 12746 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 1: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:00:23.855427 12819 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:23.872807 12824 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.872702 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:23.866199 12895 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
W20260504 20:00:23.874591 12791 consensus_peers.cc:597] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): Couldn't send request to peer 82787d8e64d24ca885ca517aa8ed1be2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260504 20:00:23.875026 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 105421us
I20260504 20:00:23.889478 12714 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:23.890728 12714 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:23.900136 12944 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: Bootstrap starting.
I20260504 20:00:23.902349 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:23.913396 13352 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:23.920094 13408 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:23.952978 13352 raft_consensus.cc:697] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:23.963689 13352 consensus_queue.cc:237] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:23.979609 13407 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:23.972812 13423 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.130s
I20260504 20:00:24.020051 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:00:24.060351 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:24.036887 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:24.065028 13425 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260504 20:00:24.065889 13387 consensus_queue.cc:1048] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260504 20:00:24.123266 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 59341.6us
I20260504 20:00:24.134697 13407 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 20:00:24.129463 13418 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.143592 12825 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:24.209213 13435 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:24.210084 13435 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:24.310221 13436 log.cc:927] Time spent T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.079s user 0.004s sys 0.000s
W20260504 20:00:24.317926 13437 log.cc:927] Time spent T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20260504 20:00:24.213757 13435 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:24.364596 13418 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:24.371263 12944 tablet_bootstrap.cc:654] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: Neither blocks nor log segments found. Creating new log.
I20260504 20:00:24.423709 12944 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: No bootstrap required, opened a new log
I20260504 20:00:24.424399 12944 ts_tablet_manager.cc:1403] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent bootstrapping tablet: real 0.532s user 0.015s sys 0.023s
I20260504 20:00:24.428452 12944 raft_consensus.cc:359] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.429313 12944 raft_consensus.cc:385] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 20:00:24.429648 12944 raft_consensus.cc:740] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Initialized, Role: FOLLOWER
I20260504 20:00:24.430402 12865 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:24.431490 13422 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 20:00:24.430460 12944 consensus_queue.cc:260] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.433612 12944 ts_tablet_manager.cc:1434] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260504 20:00:24.551116 12716 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:24.551975 13433 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 20:00:24.582315 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:24.583093 12744 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:24.601399 13331 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:24.603569 13331 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.605636 13331 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:24.628693 13449 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:24.629258 13449 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.630998 13449 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:24.659744 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:24.660399 12824 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 0.
I20260504 20:00:24.676492 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:24.677349 12744 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:00:24.678905 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:24.679636 12743 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 0.
I20260504 20:00:24.683668 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:24.693776 12863 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:24.698463 13412 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:24.698828 13412 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:24.699139 13412 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:24.702802 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:24.703724 13447 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 20:00:24.704234 13447 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:24.704764 13447 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:24.739908 13447 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.741830 13447 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:24.752220 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:00:24.752619 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 48583.2us
I20260504 20:00:24.752879 12749 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260504 20:00:24.764452 12749 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:24.765704 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:24.766552 13447 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 20:00:24.767254 13447 raft_consensus.cc:697] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:24.785507 13447 consensus_queue.cc:237] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.787132 13433 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:24.787700 13433 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.788000 13412 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:24.790156 13433 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:24.798976 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:24.806417 13409 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:24.807972 13412 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:24.810851 12644 catalog_manager.cc:5671] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:24.827315 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:24.863771 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:24.864673 12749 raft_consensus.cc:2393] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 1: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:24.868227 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:24.869053 12898 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:24.869524 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:24.870357 12896 raft_consensus.cc:2393] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:24.877184 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:24.877806 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:24.878762 12891 raft_consensus.cc:2393] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 1: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:24.879727 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:24.879884 12791 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:24.882230 13451 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 20:00:24.895449 12865 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:24.896970 13423 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 20:00:24.901986 12791 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:24.903362 13447 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:24.906178 12716 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:24.907546 13351 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:24.910516 13351 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:24.930671 12645 catalog_manager.cc:5671] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:00:25.091249 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:25.232191 13418 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:25.292843 13451 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 20:00:25.567361 12846 debug-util.cc:398] Leaking SignalData structure 0x7b0800157ac0 after lost signal to thread 11539
W20260504 20:00:25.568480 12846 debug-util.cc:398] Leaking SignalData structure 0x7b080015e160 after lost signal to thread 12682
W20260504 20:00:25.569752 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002a70c0 after lost signal to thread 12774
W20260504 20:00:25.570891 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002867e0 after lost signal to thread 12849
W20260504 20:00:25.571936 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002d5100 after lost signal to thread 12923
W20260504 20:00:26.140491 12696 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260504 20:00:26.176470 13412 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:26.177557 13412 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:26.180325 13412 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:26.181483 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:26.182578 12822 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:26.183924 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:26.184787 12742 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:26.183969 12863 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:26.187077 13412 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:26.187534 13412 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:26.187886 13412 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:26.197585 13412 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:26.199635 13412 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:26.200630 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:26.201145 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:26.201329 12749 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:26.201709 12817 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:26.212404 12817 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:26.213675 12863 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:26.214232 12749 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:26.215049 13470 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:26.215646 13470 raft_consensus.cc:697] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:26.216584 13470 consensus_queue.cc:237] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:26.255977 12645 catalog_manager.cc:5671] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 0 to 2, leader changed from <none> to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:26.704715 12749 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260504 20:00:26.706401 13470 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 20:00:26.734831 12817 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260504 20:00:26.742043 13470 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 20:00:27.758666 11536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 20:00:27.795763 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 20:00:27.796883 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 20:00:27.800356 13490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 20:00:27.801671 11536 server_base.cc:1061] running on GCE node
I20260504 20:00:27.805739 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260504 20:00:27.806089 11536 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260504 20:00:27.806474 11536 hybrid_clock.cc:648] HybridClock initialized: now 1777924827806450 us; error 0 us; skew 500 ppm
I20260504 20:00:27.807466 11536 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 20:00:27.811213 11536 webserver.cc:492] Webserver started at http://127.11.68.4:34675/ using document root <none> and password file <none>
I20260504 20:00:27.811794 11536 fs_manager.cc:362] Metadata directory not provided
I20260504 20:00:27.812050 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 20:00:27.812434 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 20:00:27.814350 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-3-root/instance:
uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
format_stamp: "Formatted at 2026-05-04 20:00:27 on dist-test-slave-3x8x"
I20260504 20:00:27.823807 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260504 20:00:27.828989 13495 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:27.830806 11536 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260504 20:00:27.831269 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-3-root
uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
format_stamp: "Formatted at 2026-05-04 20:00:27 on dist-test-slave-3x8x"
I20260504 20:00:27.831658 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 20:00:27.851405 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 20:00:27.853000 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 20:00:27.855098 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 20:00:27.858201 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 20:00:27.858412 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:27.858769 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 20:00:27.858937 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 20:00:28.018527 11536 rpc_server.cc:307] RPC server started. Bound to: 127.11.68.4:41009
I20260504 20:00:28.091221 13557 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.68.4:41009 every 8 connection(s)
I20260504 20:00:28.103281 11536 auto_leader_rebalancer-test.cc:417] add a tserver: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:28.116822 13558 heartbeater.cc:344] Connected to a master server at 127.11.68.62:41865
I20260504 20:00:28.117358 13558 heartbeater.cc:461] Registering TS with master...
I20260504 20:00:28.118397 13558 heartbeater.cc:507] Master 127.11.68.62:41865 requested a full tablet report, sending...
I20260504 20:00:28.120882 12645 ts_manager.cc:194] Registered new tserver with Master: ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:28.122938 12645 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36606
I20260504 20:00:28.372090 12742 consensus_queue.cc:237] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:28.383142 12895 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:28.388306 12822 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:28.392190 13351 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:28.403153 13351 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:28.418226 13562 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.420288 12891 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.425283 12822 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
W20260504 20:00:28.437175 12715 consensus_peers.cc:597] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f00885c8870d4ef98c5c0f9299c458ea. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:28.446633 12645 catalog_manager.cc:5671] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.450609 12742 consensus_queue.cc:237] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:28.591384 12891 raft_consensus.cc:1275] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:28.595918 12715 consensus_peers.cc:597] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 445dd93bcaf04bae88b84d4eb9eddb60. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:28.597370 13564 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:28.600813 12822 raft_consensus.cc:1275] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:28.602516 13562 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:28.620425 12822 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.625890 13351 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.661139 12895 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.703666 12646 catalog_manager.cc:5671] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.783222 13584 ts_tablet_manager.cc:933] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:28.805960 13584 tablet_copy_client.cc:323] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:28.855170 12825 consensus_queue.cc:237] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:28.881704 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet f00885c8870d4ef98c5c0f9299c458ea from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:28.882721 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet f00885c8870d4ef98c5c0f9299c458ea from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-f00885c8870d4ef98c5c0f9299c458ea
I20260504 20:00:28.893676 12760 tablet_copy_source_session.cc:215] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:28.897001 12895 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:28.899060 13584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00885c8870d4ef98c5c0f9299c458ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:28.901120 13473 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:28.906023 12742 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:28.920540 13473 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260504 20:00:28.942824 12790 consensus_peers.cc:597] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aada4371a0e5404bb6b69e5a2f3b1164. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:28.944680 13565 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.966598 13584 tablet_copy_client.cc:806] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:28.968829 13584 tablet_copy_client.cc:670] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:28.970548 12744 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.976231 13584 tablet_copy_client.cc:538] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:28.970441 12895 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:28.995555 12896 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:28.998170 13584 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:28.999576 12646 catalog_manager.cc:5671] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:29.032558 12744 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:29.041749 13588 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:29.046128 12822 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:29.050073 13588 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:29.085891 13470 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:29.088831 12822 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:29.134972 13558 heartbeater.cc:499] Master 127.11.68.62:41865 was elected leader, sending a full tablet report...
W20260504 20:00:29.137503 12864 consensus_peers.cc:597] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04d76f164cad4ed0bc8e39ce2403623f. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:29.139787 12644 catalog_manager.cc:5671] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:29.174373 12749 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:29.189026 13599 ts_tablet_manager.cc:933] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:29.189689 13584 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:29.197779 13584 tablet_bootstrap.cc:492] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:29.198737 13584 ts_tablet_manager.cc:1403] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.201s user 0.058s sys 0.030s
I20260504 20:00:29.203018 13584 raft_consensus.cc:359] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:29.205626 13584 raft_consensus.cc:740] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:29.206791 13584 consensus_queue.cc:260] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:29.213070 13599 tablet_copy_client.cc:323] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:29.215247 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet 445dd93bcaf04bae88b84d4eb9eddb60 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:29.215817 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet 445dd93bcaf04bae88b84d4eb9eddb60 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-445dd93bcaf04bae88b84d4eb9eddb60
I20260504 20:00:29.219970 13584 ts_tablet_manager.cc:1434] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.021s user 0.006s sys 0.006s
I20260504 20:00:29.223567 12759 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-f00885c8870d4ef98c5c0f9299c458ea received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:29.224107 12759 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-f00885c8870d4ef98c5c0f9299c458ea on tablet f00885c8870d4ef98c5c0f9299c458ea with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:29.229050 12760 tablet_copy_source_session.cc:215] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:29.236402 13599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445dd93bcaf04bae88b84d4eb9eddb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:29.267294 13599 tablet_copy_client.cc:806] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:29.267895 13599 tablet_copy_client.cc:670] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:29.272010 13599 tablet_copy_client.cc:538] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:29.282037 13599 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:29.320966 13599 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:29.322145 13599 tablet_bootstrap.cc:492] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:29.323020 13599 ts_tablet_manager.cc:1403] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.041s user 0.020s sys 0.012s
I20260504 20:00:29.326392 13599 raft_consensus.cc:359] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:29.327425 13599 raft_consensus.cc:740] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:29.328179 13599 consensus_queue.cc:260] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:29.339844 13599 ts_tablet_manager.cc:1434] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260504 20:00:29.341881 12760 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-445dd93bcaf04bae88b84d4eb9eddb60 received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:29.342378 12760 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-445dd93bcaf04bae88b84d4eb9eddb60 on tablet 445dd93bcaf04bae88b84d4eb9eddb60 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:29.416994 13599 ts_tablet_manager.cc:933] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:29.419111 13533 raft_consensus.cc:1217] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:29.425704 13599 tablet_copy_client.cc:323] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.3:39151
I20260504 20:00:29.474601 12909 tablet_copy_service.cc:140] P 732c3560a688494196d22a1a48f89968: Received BeginTabletCopySession request for tablet 04d76f164cad4ed0bc8e39ce2403623f from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:38968)
I20260504 20:00:29.475271 12909 tablet_copy_service.cc:161] P 732c3560a688494196d22a1a48f89968: Beginning new tablet copy session on tablet 04d76f164cad4ed0bc8e39ce2403623f from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:38968: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-04d76f164cad4ed0bc8e39ce2403623f
I20260504 20:00:29.486670 13584 ts_tablet_manager.cc:933] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:29.492413 12909 tablet_copy_source_session.cc:215] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:29.499930 13599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04d76f164cad4ed0bc8e39ce2403623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:29.501422 13584 tablet_copy_client.cc:323] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.2:41415
I20260504 20:00:29.526793 13599 tablet_copy_client.cc:806] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:29.527675 13599 tablet_copy_client.cc:670] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:29.531890 13599 tablet_copy_client.cc:538] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:29.548295 13599 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:29.567243 12835 tablet_copy_service.cc:140] P 5ff0ef72ec3b4667893a0420d3c7f923: Received BeginTabletCopySession request for tablet aada4371a0e5404bb6b69e5a2f3b1164 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:48774)
I20260504 20:00:29.567970 12835 tablet_copy_service.cc:161] P 5ff0ef72ec3b4667893a0420d3c7f923: Beginning new tablet copy session on tablet aada4371a0e5404bb6b69e5a2f3b1164 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:48774: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-aada4371a0e5404bb6b69e5a2f3b1164
I20260504 20:00:29.578259 12835 tablet_copy_source_session.cc:215] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:29.582840 13584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aada4371a0e5404bb6b69e5a2f3b1164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:29.604627 13599 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:29.605880 13599 tablet_bootstrap.cc:492] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:29.606619 13599 ts_tablet_manager.cc:1403] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.059s user 0.019s sys 0.010s
I20260504 20:00:29.610427 13599 raft_consensus.cc:359] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:29.611347 13599 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:29.612171 13599 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:29.614995 13599 ts_tablet_manager.cc:1434] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260504 20:00:29.616484 13533 raft_consensus.cc:1217] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:29.618587 12909 tablet_copy_service.cc:342] P 732c3560a688494196d22a1a48f89968: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-04d76f164cad4ed0bc8e39ce2403623f received from {username='slave'} at 127.0.0.1:38968
I20260504 20:00:29.619107 12909 tablet_copy_service.cc:434] P 732c3560a688494196d22a1a48f89968: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-04d76f164cad4ed0bc8e39ce2403623f on tablet 04d76f164cad4ed0bc8e39ce2403623f with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:29.638116 13584 tablet_copy_client.cc:806] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:29.640631 13584 tablet_copy_client.cc:670] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:29.649761 13584 tablet_copy_client.cc:538] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:29.667029 13584 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:29.722350 13584 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:29.725129 13584 tablet_bootstrap.cc:492] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:29.725966 13584 ts_tablet_manager.cc:1403] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.059s user 0.024s sys 0.002s
I20260504 20:00:29.729624 13584 raft_consensus.cc:359] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:29.730686 13584 raft_consensus.cc:740] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:29.731387 13584 consensus_queue.cc:260] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:29.740778 13584 ts_tablet_manager.cc:1434] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260504 20:00:29.751348 12835 tablet_copy_service.cc:342] P 5ff0ef72ec3b4667893a0420d3c7f923: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-aada4371a0e5404bb6b69e5a2f3b1164 received from {username='slave'} at 127.0.0.1:48774
I20260504 20:00:29.751927 12835 tablet_copy_service.cc:434] P 5ff0ef72ec3b4667893a0420d3c7f923: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-aada4371a0e5404bb6b69e5a2f3b1164 on tablet aada4371a0e5404bb6b69e5a2f3b1164 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:29.894966 13533 raft_consensus.cc:1217] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:29.950717 13351 raft_consensus.cc:1064] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:29.952907 13351 consensus_queue.cc:237] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:29.982107 13533 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:29.982889 12896 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:29.984216 13351 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:29.986532 13351 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:29.990449 12818 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:29.997215 13581 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:29.998607 13562 raft_consensus.cc:1064] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:30.006891 13562 consensus_queue.cc:237] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:30.018183 13581 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.027187 13533 raft_consensus.cc:1217] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:30.037829 13533 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.053758 12891 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.074550 12644 catalog_manager.cc:5671] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.100548 12893 raft_consensus.cc:1275] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:30.124059 12817 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.134639 12822 raft_consensus.cc:1275] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:30.134035 13614 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260504 20:00:30.156606 13533 raft_consensus.cc:1275] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:30.168766 13562 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260504 20:00:30.188660 13351 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260504 20:00:30.193781 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 45062.7us
I20260504 20:00:30.233107 13616 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.239205 13533 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.249432 12893 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.260866 12818 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.268702 12648 catalog_manager.cc:5671] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.278120 13485 raft_consensus.cc:1064] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:30.281205 13485 consensus_queue.cc:237] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:30.296363 12742 consensus_queue.cc:237] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:30.302631 13533 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:30.304430 13485 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:30.304997 12744 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:30.306532 12891 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:30.309224 13621 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:30.309922 13532 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:30.311313 13621 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:30.313301 12818 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:30.313894 13622 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:30.315254 13562 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
W20260504 20:00:30.472597 13630 log.cc:927] Time spent T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.094s user 0.000s sys 0.003s
W20260504 20:00:30.478051 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 86287us
I20260504 20:00:30.511729 13631 raft_consensus.cc:1064] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:30.510638 13351 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.524582 13631 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:00:30.530429 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 111199us
W20260504 20:00:30.542997 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 59137us
I20260504 20:00:30.545161 13533 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.552841 12743 consensus_queue.cc:237] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:30.565490 12895 raft_consensus.cc:1275] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:30.579165 13582 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260504 20:00:30.587908 12817 raft_consensus.cc:2955] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.578111 13565 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.606010 13532 raft_consensus.cc:1275] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:30.609064 13616 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:30.618197 12644 catalog_manager.cc:5671] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:30.627971 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet f00885c8870d4ef98c5c0f9299c458ea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260504 20:00:30.630963 12635 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 63343.6us
I20260504 20:00:30.646862 13582 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.651331 13531 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.656889 13532 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:30.694613 12646 catalog_manager.cc:5671] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:30.730932 12883 tablet_service.cc:1558] Processing DeleteTablet for tablet f00885c8870d4ef98c5c0f9299c458ea with delete_type TABLET_DATA_TOMBSTONED (TS 732c3560a688494196d22a1a48f89968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37306
W20260504 20:00:30.868309 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 297310us
I20260504 20:00:30.927240 13533 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:30.938732 13566 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
W20260504 20:00:30.980336 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 103141us
W20260504 20:00:30.987344 13646 log.cc:927] Time spent T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.126s user 0.001s sys 0.006s
W20260504 20:00:31.013285 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 411821us
I20260504 20:00:31.018143 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 445dd93bcaf04bae88b84d4eb9eddb60 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260504 20:00:31.020648 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 150870us
I20260504 20:00:31.079456 12891 raft_consensus.cc:2955] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.073871 12896 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.145728 13648 tablet_replica.cc:333] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: stopping tablet replica
W20260504 20:00:31.173044 12863 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.009s sys 0.018s
W20260504 20:00:31.173549 12863 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.009s sys 0.018s
W20260504 20:00:31.174160 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 612192us
I20260504 20:00:31.179065 12644 catalog_manager.cc:5671] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:31.188030 12742 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.221830 12825 consensus_queue.cc:237] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:31.230770 12817 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:31.234246 12749 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:31.260852 13648 raft_consensus.cc:2243] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:31.266127 13648 raft_consensus.cc:2272] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:31.288517 13531 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:31.289861 13644 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:31.294592 13643 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260504 20:00:31.297585 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 113779us
I20260504 20:00:31.303890 13638 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.314239 13531 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.323282 13648 ts_tablet_manager.cc:1916] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:31.341154 13644 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:31.357439 12644 catalog_manager.cc:5671] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:31.385161 12735 tablet_service.cc:1558] Processing DeleteTablet for tablet aada4371a0e5404bb6b69e5a2f3b1164 with delete_type TABLET_DATA_TOMBSTONED (TS 82787d8e64d24ca885ca517aa8ed1be2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57450
I20260504 20:00:31.403398 13653 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.417768 12742 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.431016 12633 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet aada4371a0e5404bb6b69e5a2f3b1164 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:31.441416 12806 tablet_service.cc:1558] Processing DeleteTablet for tablet 445dd93bcaf04bae88b84d4eb9eddb60 with delete_type TABLET_DATA_TOMBSTONED (TS 5ff0ef72ec3b4667893a0420d3c7f923 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53094
I20260504 20:00:31.464624 13659 tablet_replica.cc:333] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: stopping tablet replica
I20260504 20:00:31.466466 13659 raft_consensus.cc:2243] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:31.467713 13659 raft_consensus.cc:2272] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:31.470712 12893 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:31.486263 13638 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:31.491312 12644 catalog_manager.cc:5671] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.491281 13659 ts_tablet_manager.cc:1916] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:31.497732 13661 tablet_replica.cc:333] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: stopping tablet replica
I20260504 20:00:31.510535 13661 raft_consensus.cc:2243] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:31.502344 13531 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.517387 12818 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.516638 13648 ts_tablet_manager.cc:1929] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:31.529078 13648 log.cc:1199] T f00885c8870d4ef98c5c0f9299c458ea P 732c3560a688494196d22a1a48f89968: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/wals/f00885c8870d4ef98c5c0f9299c458ea
I20260504 20:00:31.514797 12897 raft_consensus.cc:2955] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.540639 13661 raft_consensus.cc:2272] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:31.547422 12635 catalog_manager.cc:5002] TS 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): tablet f00885c8870d4ef98c5c0f9299c458ea (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:31.562114 12896 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:31.579476 12825 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:31.581248 13661 ts_tablet_manager.cc:1916] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:31.582430 13531 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:31.582643 13641 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:31.585316 13641 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:31.593163 13659 ts_tablet_manager.cc:1929] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:31.593775 13659 log.cc:1199] T aada4371a0e5404bb6b69e5a2f3b1164 P 82787d8e64d24ca885ca517aa8ed1be2: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/wals/aada4371a0e5404bb6b69e5a2f3b1164
I20260504 20:00:31.592999 13653 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.595883 12635 catalog_manager.cc:5002] TS 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): tablet aada4371a0e5404bb6b69e5a2f3b1164 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:31.614363 13531 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.629110 12822 raft_consensus.cc:2955] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:31.647461 13661 ts_tablet_manager.cc:1929] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:31.648262 13661 log.cc:1199] T 445dd93bcaf04bae88b84d4eb9eddb60 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/wals/445dd93bcaf04bae88b84d4eb9eddb60
I20260504 20:00:31.658298 12633 catalog_manager.cc:5002] TS 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): tablet 445dd93bcaf04bae88b84d4eb9eddb60 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:31.664495 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 04d76f164cad4ed0bc8e39ce2403623f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:31.703012 12644 catalog_manager.cc:5671] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:31.718914 12735 tablet_service.cc:1558] Processing DeleteTablet for tablet 04d76f164cad4ed0bc8e39ce2403623f with delete_type TABLET_DATA_TOMBSTONED (TS 82787d8e64d24ca885ca517aa8ed1be2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57450
I20260504 20:00:31.719964 13659 tablet_replica.cc:333] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: stopping tablet replica
I20260504 20:00:31.742909 13659 raft_consensus.cc:2243] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:31.743796 13659 raft_consensus.cc:2272] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:31.747223 13659 ts_tablet_manager.cc:1916] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:31.769269 13659 ts_tablet_manager.cc:1929] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:31.769714 13659 log.cc:1199] T 04d76f164cad4ed0bc8e39ce2403623f P 82787d8e64d24ca885ca517aa8ed1be2: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/wals/04d76f164cad4ed0bc8e39ce2403623f
I20260504 20:00:31.771412 12635 catalog_manager.cc:5002] TS 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): tablet 04d76f164cad4ed0bc8e39ce2403623f (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:32.765502 12825 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:32.773934 12894 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:32.775465 12790 consensus_peers.cc:597] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd0471a7f3374ceab25bb10c891e18e8. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:32.775887 13663 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:32.777593 12749 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:32.779129 13663 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:32.795660 13638 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:32.798890 12749 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:32.813200 12742 consensus_queue.cc:237] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:32.820261 12897 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:32.820899 12646 catalog_manager.cc:5671] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:32.891934 12897 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:32.894165 13582 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260504 20:00:32.901871 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 92778.2us
W20260504 20:00:32.909086 12715 consensus_peers.cc:597] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4472f9104c524735bcfeed8474933a75. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:32.911777 12825 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:32.916419 13670 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:32.944591 13670 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:32.949302 12894 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:32.951290 12822 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:32.964572 12894 consensus_queue.cc:237] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:32.968112 12645 catalog_manager.cc:5671] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:33.012104 12822 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:33.012969 12742 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:33.014319 12864 consensus_peers.cc:597] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe81ef9fbbe44f89a19e923171496fa5. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:33.016072 13653 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:33.017220 13641 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:33.028208 13653 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:33.031021 12822 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:33.044418 12745 consensus_queue.cc:237] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.030937 12742 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:33.049996 12644 catalog_manager.cc:5671] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:33.086833 12894 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:33.088631 12715 consensus_peers.cc:597] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6673c1682edf457ba633ba714a47587f. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:33.090030 13670 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:33.106184 12822 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:33.106417 13670 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:33.112326 12894 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:33.121650 13680 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:33.138833 12646 catalog_manager.cc:5671] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:33.148061 12825 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:33.336305 13683 ts_tablet_manager.cc:933] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:33.340795 13683 tablet_copy_client.cc:323] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.2:41415
I20260504 20:00:33.342566 12835 tablet_copy_service.cc:140] P 5ff0ef72ec3b4667893a0420d3c7f923: Received BeginTabletCopySession request for tablet fd0471a7f3374ceab25bb10c891e18e8 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:48774)
I20260504 20:00:33.343169 12835 tablet_copy_service.cc:161] P 5ff0ef72ec3b4667893a0420d3c7f923: Beginning new tablet copy session on tablet fd0471a7f3374ceab25bb10c891e18e8 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:48774: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-fd0471a7f3374ceab25bb10c891e18e8
I20260504 20:00:33.350684 12835 tablet_copy_source_session.cc:215] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:33.354566 13683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0471a7f3374ceab25bb10c891e18e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:33.387686 13683 tablet_copy_client.cc:806] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:33.388516 13683 tablet_copy_client.cc:670] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:33.389967 13685 ts_tablet_manager.cc:933] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:33.396987 13683 tablet_copy_client.cc:538] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:33.416709 13685 tablet_copy_client.cc:323] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:33.420099 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet 4472f9104c524735bcfeed8474933a75 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:33.420702 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet 4472f9104c524735bcfeed8474933a75 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-4472f9104c524735bcfeed8474933a75
I20260504 20:00:33.421494 13683 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:33.428617 12760 tablet_copy_source_session.cc:215] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:33.442246 13685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4472f9104c524735bcfeed8474933a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:33.456398 13683 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:33.457662 13683 tablet_bootstrap.cc:492] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:33.458590 13683 ts_tablet_manager.cc:1403] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.004s
I20260504 20:00:33.462147 13683 raft_consensus.cc:359] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.463275 13683 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:33.464111 13683 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.467514 13685 tablet_copy_client.cc:806] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:33.468251 13685 tablet_copy_client.cc:670] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:33.476270 13683 ts_tablet_manager.cc:1434] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.017s user 0.012s sys 0.000s
I20260504 20:00:33.488979 12835 tablet_copy_service.cc:342] P 5ff0ef72ec3b4667893a0420d3c7f923: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-fd0471a7f3374ceab25bb10c891e18e8 received from {username='slave'} at 127.0.0.1:48774
I20260504 20:00:33.489470 12835 tablet_copy_service.cc:434] P 5ff0ef72ec3b4667893a0420d3c7f923: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-fd0471a7f3374ceab25bb10c891e18e8 on tablet fd0471a7f3374ceab25bb10c891e18e8 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:33.496614 13683 ts_tablet_manager.cc:933] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:33.516430 13685 tablet_copy_client.cc:538] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:33.557411 13685 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:33.566191 13683 tablet_copy_client.cc:323] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.3:39151
I20260504 20:00:33.570603 12909 tablet_copy_service.cc:140] P 732c3560a688494196d22a1a48f89968: Received BeginTabletCopySession request for tablet fe81ef9fbbe44f89a19e923171496fa5 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:38968)
I20260504 20:00:33.571209 12909 tablet_copy_service.cc:161] P 732c3560a688494196d22a1a48f89968: Beginning new tablet copy session on tablet fe81ef9fbbe44f89a19e923171496fa5 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:38968: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-fe81ef9fbbe44f89a19e923171496fa5
I20260504 20:00:33.580658 12909 tablet_copy_source_session.cc:215] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:33.584618 13683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe81ef9fbbe44f89a19e923171496fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:33.603178 13689 ts_tablet_manager.cc:933] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:33.609298 13683 tablet_copy_client.cc:806] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:33.610124 13683 tablet_copy_client.cc:670] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:33.612363 13689 tablet_copy_client.cc:323] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:33.613265 13685 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:33.614373 13683 tablet_copy_client.cc:538] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:33.615051 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet 6673c1682edf457ba633ba714a47587f from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:33.614421 13685 tablet_bootstrap.cc:492] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:33.615638 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet 6673c1682edf457ba633ba714a47587f from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-6673c1682edf457ba633ba714a47587f
I20260504 20:00:33.616192 13685 ts_tablet_manager.cc:1403] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.059s user 0.025s sys 0.008s
I20260504 20:00:33.620489 13685 raft_consensus.cc:359] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.621614 13685 raft_consensus.cc:740] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:33.622586 13685 consensus_queue.cc:260] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.624799 13683 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:33.625646 13685 ts_tablet_manager.cc:1434] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260504 20:00:33.633667 12759 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-4472f9104c524735bcfeed8474933a75 received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:33.634213 12759 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-4472f9104c524735bcfeed8474933a75 on tablet 4472f9104c524735bcfeed8474933a75 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:33.648811 13683 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:33.649951 13683 tablet_bootstrap.cc:492] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:33.650744 13683 ts_tablet_manager.cc:1403] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20260504 20:00:33.653888 13683 raft_consensus.cc:359] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.654786 13683 raft_consensus.cc:740] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:33.655484 13683 consensus_queue.cc:260] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.659258 13683 ts_tablet_manager.cc:1434] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260504 20:00:33.662155 12909 tablet_copy_service.cc:342] P 732c3560a688494196d22a1a48f89968: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-fe81ef9fbbe44f89a19e923171496fa5 received from {username='slave'} at 127.0.0.1:38968
I20260504 20:00:33.662660 12909 tablet_copy_service.cc:434] P 732c3560a688494196d22a1a48f89968: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-fe81ef9fbbe44f89a19e923171496fa5 on tablet fe81ef9fbbe44f89a19e923171496fa5 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:33.679960 12760 tablet_copy_source_session.cc:215] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:33.687903 13689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6673c1682edf457ba633ba714a47587f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:33.719468 13689 tablet_copy_client.cc:806] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:33.720320 13689 tablet_copy_client.cc:670] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:33.726559 13689 tablet_copy_client.cc:538] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:33.738214 13689 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:33.762674 13531 raft_consensus.cc:1217] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:33.781504 13689 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:33.782663 13689 tablet_bootstrap.cc:492] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:33.783568 13689 ts_tablet_manager.cc:1403] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.046s user 0.023s sys 0.000s
I20260504 20:00:33.787254 13689 raft_consensus.cc:359] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.788527 13689 raft_consensus.cc:740] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:33.789378 13689 consensus_queue.cc:260] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:33.804445 13689 ts_tablet_manager.cc:1434] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260504 20:00:33.807272 12760 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-6673c1682edf457ba633ba714a47587f received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:33.807771 12760 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-6673c1682edf457ba633ba714a47587f on tablet 6673c1682edf457ba633ba714a47587f with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:33.822618 13531 raft_consensus.cc:1217] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:34.031039 13531 raft_consensus.cc:1217] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:34.167110 13643 raft_consensus.cc:1064] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:34.169850 13643 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:34.187599 13531 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:34.193038 12745 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:34.193497 13673 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:34.195521 12897 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:34.197693 13531 raft_consensus.cc:1217] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:34.208855 13666 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:34.214413 13673 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:34.255496 13666 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.260223 12745 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.264727 12894 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.270182 13532 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.288717 12646 catalog_manager.cc:5671] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.357230 12825 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:34.378444 13532 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:34.380084 13673 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:34.405525 12742 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:34.413048 13680 raft_consensus.cc:1064] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:34.416029 13680 consensus_queue.cc:237] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:34.415731 13643 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.425473 13531 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.437073 13698 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:34.461613 12742 raft_consensus.cc:2955] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.467799 12645 catalog_manager.cc:5671] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.474476 12633 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet fd0471a7f3374ceab25bb10c891e18e8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:34.506162 12883 tablet_service.cc:1558] Processing DeleteTablet for tablet fd0471a7f3374ceab25bb10c891e18e8 with delete_type TABLET_DATA_TOMBSTONED (TS 732c3560a688494196d22a1a48f89968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37306
I20260504 20:00:34.525794 13708 tablet_replica.cc:333] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: stopping tablet replica
I20260504 20:00:34.527972 13708 raft_consensus.cc:2243] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:34.529253 13708 raft_consensus.cc:2272] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:34.550349 13708 ts_tablet_manager.cc:1916] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:34.559434 12894 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:34.561995 13531 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:34.570495 13582 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:34.573419 13582 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260504 20:00:34.604490 13653 raft_consensus.cc:1064] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:34.620771 12822 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:34.608801 13653 consensus_queue.cc:237] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:34.624490 13670 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:34.648777 13670 raft_consensus.cc:1064] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:34.667708 13670 consensus_queue.cc:237] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:34.673334 13680 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.678138 12822 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:34.687361 13708 ts_tablet_manager.cc:1929] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:34.715369 13708 log.cc:1199] T fd0471a7f3374ceab25bb10c891e18e8 P 732c3560a688494196d22a1a48f89968: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/wals/fd0471a7f3374ceab25bb10c891e18e8
W20260504 20:00:35.037106 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 352482us
W20260504 20:00:35.276309 13722 log.cc:927] Time spent T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260504 20:00:35.288004 12635 catalog_manager.cc:5002] TS 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): tablet fd0471a7f3374ceab25bb10c891e18e8 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:35.305114 12817 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260504 20:00:35.328565 12822 consensus_meta.cc:287] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.624s user 0.012s sys 0.000s
I20260504 20:00:35.354200 13716 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260504 20:00:35.394145 12892 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:35.415246 12894 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:35.428529 12816 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:35.431710 13531 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260504 20:00:35.466995 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 44591.3us
I20260504 20:00:35.469386 13670 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:35.405872 13532 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:35.482160 13735 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
W20260504 20:00:35.514544 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 52265.8us
W20260504 20:00:35.576062 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 102096us
I20260504 20:00:35.581959 12743 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:35.580227 12645 catalog_manager.cc:5671] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:35.593415 13719 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:35.722600 13731 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:35.723510 13731 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:35.727211 13731 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:35.733039 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:35.747187 13718 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:35.756783 13739 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:35.776521 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:35.803740 12897 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:35.816035 13533 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:00:35.907867 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 73051.5us
I20260504 20:00:35.942301 12716 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:35.942780 13721 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:35.949329 13531 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:35.994587 13727 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:00:35.994967 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 53181us
I20260504 20:00:36.002789 12825 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:35.989434 12818 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.020038 12644 catalog_manager.cc:5671] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.070726 12648 catalog_manager.cc:5671] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:36.076967 12749 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.122308 12898 consensus_queue.cc:237] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:36.149633 13533 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:36.151669 13714 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:36.158380 12825 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:36.161460 12749 consensus_queue.cc:237] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:36.162840 13723 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260504 20:00:36.160821 13729 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.165709 12748 consensus_queue.cc:237] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:36.171591 13531 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.172735 12825 raft_consensus.cc:2955] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.187023 13533 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:36.188539 13727 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:36.205034 12898 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:36.208068 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet fe81ef9fbbe44f89a19e923171496fa5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:36.210626 12645 catalog_manager.cc:5671] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.214133 13706 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.224423 13532 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:36.227207 13680 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:36.232853 12735 tablet_service.cc:1558] Processing DeleteTablet for tablet fe81ef9fbbe44f89a19e923171496fa5 with delete_type TABLET_DATA_TOMBSTONED (TS 82787d8e64d24ca885ca517aa8ed1be2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57450
I20260504 20:00:36.227171 13533 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.246280 13680 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260504 20:00:36.264367 12893 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:36.284710 13754 tablet_replica.cc:333] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: stopping tablet replica
I20260504 20:00:36.286768 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 6673c1682edf457ba633ba714a47587f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:36.287772 13754 raft_consensus.cc:2243] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:36.288717 13754 raft_consensus.cc:2272] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:36.293684 13754 ts_tablet_manager.cc:1916] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:36.293078 12647 catalog_manager.cc:5671] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:36.294201 12892 raft_consensus.cc:2955] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.288179 13706 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.316545 13754 ts_tablet_manager.cc:1929] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:36.317301 13754 log.cc:1199] T fe81ef9fbbe44f89a19e923171496fa5 P 82787d8e64d24ca885ca517aa8ed1be2: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/wals/fe81ef9fbbe44f89a19e923171496fa5
I20260504 20:00:36.322046 12806 tablet_service.cc:1558] Processing DeleteTablet for tablet 6673c1682edf457ba633ba714a47587f with delete_type TABLET_DATA_TOMBSTONED (TS 5ff0ef72ec3b4667893a0420d3c7f923 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53094
I20260504 20:00:36.322415 13533 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.331678 13727 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:36.336326 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 4472f9104c524735bcfeed8474933a75 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:36.343544 12645 catalog_manager.cc:5671] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.346530 12635 catalog_manager.cc:5002] TS 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): tablet fe81ef9fbbe44f89a19e923171496fa5 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:36.369896 12896 raft_consensus.cc:2955] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:36.376850 12814 tablet_service.cc:1558] Processing DeleteTablet for tablet 4472f9104c524735bcfeed8474933a75 with delete_type TABLET_DATA_TOMBSTONED (TS 5ff0ef72ec3b4667893a0420d3c7f923 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53094
I20260504 20:00:36.390722 13755 tablet_replica.cc:333] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: stopping tablet replica
I20260504 20:00:36.392935 13755 raft_consensus.cc:2243] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:36.394937 13755 raft_consensus.cc:2272] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:36.400022 13755 ts_tablet_manager.cc:1916] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:36.423133 13755 ts_tablet_manager.cc:1929] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:36.423629 13755 log.cc:1199] T 6673c1682edf457ba633ba714a47587f P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/wals/6673c1682edf457ba633ba714a47587f
I20260504 20:00:36.425757 13755 tablet_replica.cc:333] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: stopping tablet replica
I20260504 20:00:36.426406 12633 catalog_manager.cc:5002] TS 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): tablet 6673c1682edf457ba633ba714a47587f (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:36.427058 13755 raft_consensus.cc:2243] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:36.428213 13755 raft_consensus.cc:2272] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:36.431893 13755 ts_tablet_manager.cc:1916] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:36.451565 13755 ts_tablet_manager.cc:1929] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:36.452138 13755 log.cc:1199] T 4472f9104c524735bcfeed8474933a75 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/wals/4472f9104c524735bcfeed8474933a75
I20260504 20:00:36.457846 12633 catalog_manager.cc:5002] TS 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): tablet 4472f9104c524735bcfeed8474933a75 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
W20260504 20:00:36.564224 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002d57a0 after lost signal to thread 11539
W20260504 20:00:36.565490 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002658a0 after lost signal to thread 12682
W20260504 20:00:36.566818 12846 debug-util.cc:398] Leaking SignalData structure 0x7b080015e100 after lost signal to thread 12774
W20260504 20:00:36.567938 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002d5960 after lost signal to thread 12849
W20260504 20:00:36.569000 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08001bcca0 after lost signal to thread 12923
W20260504 20:00:36.570626 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002e7d00 after lost signal to thread 13557
I20260504 20:00:37.470579 12825 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
W20260504 20:00:37.478665 12790 consensus_peers.cc:597] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e115d71520b4beb9b2984dd586354e4. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:37.479004 12898 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:37.480046 12749 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:37.481317 13756 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:37.482488 13698 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:37.491324 13756 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.493098 12898 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.498687 12749 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.512625 12742 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:37.521670 12647 catalog_manager.cc:5671] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.620492 12825 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:37.620802 12715 consensus_peers.cc:597] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 756858b25f1f4125bf837cb8379e3b02. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:37.624385 13726 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:37.639427 12894 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:37.641733 13726 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:37.645112 13765 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.647688 12825 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.678468 12742 consensus_queue.cc:237] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:37.688143 12645 catalog_manager.cc:5671] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.698338 12896 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
W20260504 20:00:37.708701 12715 consensus_peers.cc:597] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00b8418b3a8441ee845a0591a75a9f17. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:37.709054 12894 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:37.709916 12816 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:37.711982 13726 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:37.714025 13738 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:37.759714 13738 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.761385 12894 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.771076 12818 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.786051 12816 consensus_queue.cc:237] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:37.804476 12646 catalog_manager.cc:5671] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.844784 12893 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:37.844893 12742 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:37.846230 12790 consensus_peers.cc:597] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5409cb132bc14d1cafaa29b22ced9dc9. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:37.847625 13737 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:37.849732 13737 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:37.863062 13737 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.866614 12742 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.871969 12894 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.902823 12645 catalog_manager.cc:5671] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:37.991007 13776 ts_tablet_manager.cc:933] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:37.999771 13776 tablet_copy_client.cc:323] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.2:41415
I20260504 20:00:38.001900 12835 tablet_copy_service.cc:140] P 5ff0ef72ec3b4667893a0420d3c7f923: Received BeginTabletCopySession request for tablet 1e115d71520b4beb9b2984dd586354e4 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:48774)
I20260504 20:00:38.002490 12835 tablet_copy_service.cc:161] P 5ff0ef72ec3b4667893a0420d3c7f923: Beginning new tablet copy session on tablet 1e115d71520b4beb9b2984dd586354e4 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:48774: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-1e115d71520b4beb9b2984dd586354e4
I20260504 20:00:38.014604 12835 tablet_copy_source_session.cc:215] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:38.020622 13776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e115d71520b4beb9b2984dd586354e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:38.018461 13778 ts_tablet_manager.cc:933] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:38.036212 13778 tablet_copy_client.cc:323] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:38.047108 13776 tablet_copy_client.cc:806] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:38.048027 13776 tablet_copy_client.cc:670] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:38.052771 13776 tablet_copy_client.cc:538] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:38.053063 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet 756858b25f1f4125bf837cb8379e3b02 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:38.054239 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet 756858b25f1f4125bf837cb8379e3b02 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-756858b25f1f4125bf837cb8379e3b02
I20260504 20:00:38.065441 12760 tablet_copy_source_session.cc:215] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:38.071843 13778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756858b25f1f4125bf837cb8379e3b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:38.087910 13776 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:38.102083 13778 tablet_copy_client.cc:806] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:38.102890 13778 tablet_copy_client.cc:670] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:38.108971 13778 tablet_copy_client.cc:538] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:38.121179 13778 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:38.123050 13781 ts_tablet_manager.cc:933] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:38.134253 13776 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:38.135438 13776 tablet_bootstrap.cc:492] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:38.136415 13776 ts_tablet_manager.cc:1403] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.049s user 0.027s sys 0.006s
I20260504 20:00:38.137326 13781 tablet_copy_client.cc:323] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:38.142427 13776 raft_consensus.cc:359] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:38.143748 13776 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:38.144338 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet 00b8418b3a8441ee845a0591a75a9f17 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:38.144955 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet 00b8418b3a8441ee845a0591a75a9f17 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-00b8418b3a8441ee845a0591a75a9f17
I20260504 20:00:38.144671 13776 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:38.155745 13776 ts_tablet_manager.cc:1434] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.018s user 0.009s sys 0.004s
I20260504 20:00:38.157795 12835 tablet_copy_service.cc:342] P 5ff0ef72ec3b4667893a0420d3c7f923: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-1e115d71520b4beb9b2984dd586354e4 received from {username='slave'} at 127.0.0.1:48774
I20260504 20:00:38.158389 12835 tablet_copy_service.cc:434] P 5ff0ef72ec3b4667893a0420d3c7f923: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-1e115d71520b4beb9b2984dd586354e4 on tablet 1e115d71520b4beb9b2984dd586354e4 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:38.217521 12760 tablet_copy_source_session.cc:215] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:38.224913 13778 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:38.226195 13778 tablet_bootstrap.cc:492] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:38.227087 13778 ts_tablet_manager.cc:1403] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.106s user 0.049s sys 0.030s
I20260504 20:00:38.231216 13781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00b8418b3a8441ee845a0591a75a9f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:38.230822 13778 raft_consensus.cc:359] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:38.231748 13778 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:38.232481 13778 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:38.235666 13778 ts_tablet_manager.cc:1434] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260504 20:00:38.241496 12760 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-756858b25f1f4125bf837cb8379e3b02 received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:38.242080 12760 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-756858b25f1f4125bf837cb8379e3b02 on tablet 756858b25f1f4125bf837cb8379e3b02 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:38.257032 13781 tablet_copy_client.cc:806] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:38.259093 13781 tablet_copy_client.cc:670] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:38.263715 13781 tablet_copy_client.cc:538] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:38.274921 13781 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:38.314551 13781 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:38.315613 13781 tablet_bootstrap.cc:492] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:38.316453 13781 ts_tablet_manager.cc:1403] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.042s user 0.019s sys 0.012s
I20260504 20:00:38.320016 13781 raft_consensus.cc:359] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:38.320987 13781 raft_consensus.cc:740] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:38.321766 13781 consensus_queue.cc:260] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:38.324942 13781 ts_tablet_manager.cc:1434] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20260504 20:00:38.326844 12760 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-00b8418b3a8441ee845a0591a75a9f17 received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:38.327361 12760 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-00b8418b3a8441ee845a0591a75a9f17 on tablet 00b8418b3a8441ee845a0591a75a9f17 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:38.460836 13533 raft_consensus.cc:1217] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:38.471585 13781 ts_tablet_manager.cc:933] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:38.519394 13781 tablet_copy_client.cc:323] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.2:41415
I20260504 20:00:38.521488 12835 tablet_copy_service.cc:140] P 5ff0ef72ec3b4667893a0420d3c7f923: Received BeginTabletCopySession request for tablet 5409cb132bc14d1cafaa29b22ced9dc9 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:48774)
I20260504 20:00:38.522095 12835 tablet_copy_service.cc:161] P 5ff0ef72ec3b4667893a0420d3c7f923: Beginning new tablet copy session on tablet 5409cb132bc14d1cafaa29b22ced9dc9 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:48774: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-5409cb132bc14d1cafaa29b22ced9dc9
I20260504 20:00:38.529309 12835 tablet_copy_source_session.cc:215] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:38.533753 13781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5409cb132bc14d1cafaa29b22ced9dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:38.549623 13533 raft_consensus.cc:1217] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:38.556555 13781 tablet_copy_client.cc:806] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:38.557376 13781 tablet_copy_client.cc:670] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:38.568715 13781 tablet_copy_client.cc:538] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:38.579764 13781 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:38.585554 13533 raft_consensus.cc:1217] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:38.656159 13781 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:38.657146 13781 tablet_bootstrap.cc:492] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:38.658133 13781 ts_tablet_manager.cc:1403] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.079s user 0.019s sys 0.008s
I20260504 20:00:38.661736 13781 raft_consensus.cc:359] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:38.662774 13781 raft_consensus.cc:740] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:38.663640 13781 consensus_queue.cc:260] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:38.666471 13781 ts_tablet_manager.cc:1434] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260504 20:00:38.674042 12835 tablet_copy_service.cc:342] P 5ff0ef72ec3b4667893a0420d3c7f923: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-5409cb132bc14d1cafaa29b22ced9dc9 received from {username='slave'} at 127.0.0.1:48774
I20260504 20:00:38.674504 12835 tablet_copy_service.cc:434] P 5ff0ef72ec3b4667893a0420d3c7f923: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-5409cb132bc14d1cafaa29b22ced9dc9 on tablet 5409cb132bc14d1cafaa29b22ced9dc9 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:38.980046 13531 raft_consensus.cc:1217] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:38.997440 13726 raft_consensus.cc:1064] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:39.000249 13726 consensus_queue.cc:237] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:39.006556 13788 raft_consensus.cc:1064] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:39.009846 13788 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:39.015517 13737 raft_consensus.cc:1064] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:39.019160 12816 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.021406 13531 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.022727 13680 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:39.023702 12894 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.025527 13765 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:39.027359 13765 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:39.029069 13737 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:39.060302 13533 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.060303 12894 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.067977 13533 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.069851 12818 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.071209 13796 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:39.071640 12748 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.069851 12894 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.073587 13775 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:39.074026 13788 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:39.078145 13790 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:39.079020 13756 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:39.092525 13756 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:39.102524 13788 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.116222 13531 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.132440 12818 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.149298 12896 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.154853 13790 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.159391 12894 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.159391 13775 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.163129 12647 catalog_manager.cc:5671] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.169559 12742 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.186589 13531 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.208256 13533 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:00:39.220790 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 47142.4us
I20260504 20:00:39.231611 12894 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.240162 12818 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.251739 12645 catalog_manager.cc:5671] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.300209 12645 catalog_manager.cc:5671] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.308735 12748 consensus_queue.cc:237] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:39.315855 13533 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:39.317441 13790 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:39.329537 12817 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:39.336887 12894 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:39.337880 13531 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.340752 13791 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:39.343079 12749 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:39.344492 13811 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:39.347184 12894 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.335842 13790 raft_consensus.cc:2955] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.348579 12742 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:39.354664 13533 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:39.355569 12817 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:39.356144 13796 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:39.359056 13796 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:39.364140 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 00b8418b3a8441ee845a0591a75a9f17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:39.367313 12647 catalog_manager.cc:5671] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.370988 13788 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.377204 13531 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:39.379101 12817 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.380199 13811 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:39.382438 13798 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.386504 12806 tablet_service.cc:1558] Processing DeleteTablet for tablet 00b8418b3a8441ee845a0591a75a9f17 with delete_type TABLET_DATA_TOMBSTONED (TS 5ff0ef72ec3b4667893a0420d3c7f923 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53094
I20260504 20:00:39.394225 12748 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.399188 13532 raft_consensus.cc:2955] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.404954 13531 raft_consensus.cc:2955] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.422868 12647 catalog_manager.cc:5671] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.434341 12633 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 1e115d71520b4beb9b2984dd586354e4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:39.434994 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 756858b25f1f4125bf837cb8379e3b02 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:39.444449 12883 tablet_service.cc:1558] Processing DeleteTablet for tablet 1e115d71520b4beb9b2984dd586354e4 with delete_type TABLET_DATA_TOMBSTONED (TS 732c3560a688494196d22a1a48f89968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37306
I20260504 20:00:39.446539 13811 raft_consensus.cc:1064] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:39.446352 12646 catalog_manager.cc:5671] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.449443 13811 consensus_queue.cc:237] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:39.468417 12887 tablet_service.cc:1558] Processing DeleteTablet for tablet 756858b25f1f4125bf837cb8379e3b02 with delete_type TABLET_DATA_TOMBSTONED (TS 732c3560a688494196d22a1a48f89968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37306
I20260504 20:00:39.471669 13814 tablet_replica.cc:333] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: stopping tablet replica
I20260504 20:00:39.473356 13814 raft_consensus.cc:2243] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:39.474606 13814 raft_consensus.cc:2272] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:39.479431 13814 ts_tablet_manager.cc:1916] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:39.498773 13532 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.500466 13798 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:39.500917 13814 ts_tablet_manager.cc:1929] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:39.501363 13814 log.cc:1199] T 00b8418b3a8441ee845a0591a75a9f17 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/wals/00b8418b3a8441ee845a0591a75a9f17
I20260504 20:00:39.505234 12633 catalog_manager.cc:5002] TS 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): tablet 00b8418b3a8441ee845a0591a75a9f17 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:39.505862 12748 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.507202 12894 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:39.509503 13756 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:39.515504 13811 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:39.525216 13816 tablet_replica.cc:333] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: stopping tablet replica
I20260504 20:00:39.526684 13816 raft_consensus.cc:2243] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:39.527585 13816 raft_consensus.cc:2272] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:39.540633 13816 ts_tablet_manager.cc:1916] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:39.570781 13756 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.577315 13816 ts_tablet_manager.cc:1929] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:39.578233 13816 log.cc:1199] T 1e115d71520b4beb9b2984dd586354e4 P 732c3560a688494196d22a1a48f89968: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/wals/1e115d71520b4beb9b2984dd586354e4
I20260504 20:00:39.580768 13816 tablet_replica.cc:333] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: stopping tablet replica
I20260504 20:00:39.582059 12635 catalog_manager.cc:5002] TS 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): tablet 1e115d71520b4beb9b2984dd586354e4 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:39.582484 13816 raft_consensus.cc:2243] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:39.583530 13816 raft_consensus.cc:2272] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:39.585069 12749 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.587770 13816 ts_tablet_manager.cc:1916] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:39.600576 13532 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.627518 13816 ts_tablet_manager.cc:1929] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:39.627732 12647 catalog_manager.cc:5671] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:39.640605 13816 log.cc:1199] T 756858b25f1f4125bf837cb8379e3b02 P 732c3560a688494196d22a1a48f89968: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/wals/756858b25f1f4125bf837cb8379e3b02
I20260504 20:00:39.645932 12892 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.643224 12635 catalog_manager.cc:5002] TS 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): tablet 756858b25f1f4125bf837cb8379e3b02 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:39.653501 12817 consensus_queue.cc:237] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:39.663395 13532 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:39.665472 13798 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:39.679090 12894 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:39.684593 13798 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.687227 13775 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:39.690004 13532 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.697906 12894 raft_consensus.cc:2955] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.716600 12633 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 5409cb132bc14d1cafaa29b22ced9dc9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:39.723213 12645 catalog_manager.cc:5671] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:39.743153 12735 tablet_service.cc:1558] Processing DeleteTablet for tablet 5409cb132bc14d1cafaa29b22ced9dc9 with delete_type TABLET_DATA_TOMBSTONED (TS 82787d8e64d24ca885ca517aa8ed1be2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57450
I20260504 20:00:39.746749 13821 tablet_replica.cc:333] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: stopping tablet replica
I20260504 20:00:39.749516 13821 raft_consensus.cc:2243] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:39.753168 13821 raft_consensus.cc:2272] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:39.759866 13821 ts_tablet_manager.cc:1916] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:39.775331 13821 ts_tablet_manager.cc:1929] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:39.775730 13821 log.cc:1199] T 5409cb132bc14d1cafaa29b22ced9dc9 P 82787d8e64d24ca885ca517aa8ed1be2: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/wals/5409cb132bc14d1cafaa29b22ced9dc9
I20260504 20:00:39.777331 12635 catalog_manager.cc:5002] TS 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): tablet 5409cb132bc14d1cafaa29b22ced9dc9 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:40.795627 12816 consensus_queue.cc:237] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:40.801913 12748 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:40.802320 12790 consensus_peers.cc:597] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57fe021572534bbdbbaea4aa4a64017a. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:40.801970 12892 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:40.803591 13798 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:40.804351 13775 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:40.813822 12748 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.812503 13798 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.814054 12892 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.835741 12645 catalog_manager.cc:5671] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.836942 12748 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:40.872507 12816 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:40.874459 13832 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:40.885082 12896 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:40.893013 13796 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260504 20:00:40.898264 12715 consensus_peers.cc:597] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 410644ff9c0449188275a6d49ecc0c2d. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:40.904003 13832 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.907897 12892 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.908291 12816 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.921764 12748 consensus_queue.cc:237] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:40.926651 12647 catalog_manager.cc:5671] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:40.942420 12896 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:40.944896 13796 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260504 20:00:40.949636 12715 consensus_peers.cc:597] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae27dbfd0ab84c9d8c95896c0c4cf502. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:40.950276 12816 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:40.952913 13832 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:40.961479 13834 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.963475 12818 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.963515 12896 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:40.978116 12816 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:40.984848 12646 catalog_manager.cc:5671] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:41.023092 12896 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:41.024173 12790 consensus_peers.cc:597] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a80bf1f95a74ce09071a550fe15996f. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:41.025938 13811 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:41.026327 12749 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:41.027765 13798 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:41.035559 13811 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:41.037315 12896 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:41.038183 12749 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:41.055640 12646 catalog_manager.cc:5671] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:41.246179 13843 ts_tablet_manager.cc:933] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:41.249140 13843 tablet_copy_client.cc:323] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:41.250982 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet 410644ff9c0449188275a6d49ecc0c2d from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:41.251598 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet 410644ff9c0449188275a6d49ecc0c2d from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-410644ff9c0449188275a6d49ecc0c2d
I20260504 20:00:41.259033 12760 tablet_copy_source_session.cc:215] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:41.263440 13843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410644ff9c0449188275a6d49ecc0c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:41.281553 13843 tablet_copy_client.cc:806] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:41.285673 13843 tablet_copy_client.cc:670] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:41.289525 13843 tablet_copy_client.cc:538] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:41.298161 13843 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:41.321920 13843 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:41.323920 13843 tablet_bootstrap.cc:492] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:41.324710 13843 ts_tablet_manager.cc:1403] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.011s
I20260504 20:00:41.327874 13843 raft_consensus.cc:359] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:41.328588 13843 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:41.329116 13843 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:41.334604 13843 ts_tablet_manager.cc:1434] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260504 20:00:41.337270 12760 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-410644ff9c0449188275a6d49ecc0c2d received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:41.337752 12760 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-410644ff9c0449188275a6d49ecc0c2d on tablet 410644ff9c0449188275a6d49ecc0c2d with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:41.355588 13843 ts_tablet_manager.cc:933] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:41.357510 13843 tablet_copy_client.cc:323] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.2:41415
I20260504 20:00:41.359098 12835 tablet_copy_service.cc:140] P 5ff0ef72ec3b4667893a0420d3c7f923: Received BeginTabletCopySession request for tablet 57fe021572534bbdbbaea4aa4a64017a from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:48774)
I20260504 20:00:41.359606 12835 tablet_copy_service.cc:161] P 5ff0ef72ec3b4667893a0420d3c7f923: Beginning new tablet copy session on tablet 57fe021572534bbdbbaea4aa4a64017a from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:48774: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-57fe021572534bbdbbaea4aa4a64017a
I20260504 20:00:41.366235 12835 tablet_copy_source_session.cc:215] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:41.369444 13843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57fe021572534bbdbbaea4aa4a64017a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:41.384863 13843 tablet_copy_client.cc:806] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:41.385509 13843 tablet_copy_client.cc:670] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:41.389392 13843 tablet_copy_client.cc:538] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:41.399470 13843 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:41.419142 13843 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:41.419894 13843 tablet_bootstrap.cc:492] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:41.420554 13843 ts_tablet_manager.cc:1403] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20260504 20:00:41.423389 13843 raft_consensus.cc:359] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:41.424113 13843 raft_consensus.cc:740] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:41.424661 13843 consensus_queue.cc:260] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:41.427055 13843 ts_tablet_manager.cc:1434] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260504 20:00:41.435420 12835 tablet_copy_service.cc:342] P 5ff0ef72ec3b4667893a0420d3c7f923: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-57fe021572534bbdbbaea4aa4a64017a received from {username='slave'} at 127.0.0.1:48774
I20260504 20:00:41.435899 12835 tablet_copy_service.cc:434] P 5ff0ef72ec3b4667893a0420d3c7f923: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-57fe021572534bbdbbaea4aa4a64017a on tablet 57fe021572534bbdbbaea4aa4a64017a with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:41.521191 13843 ts_tablet_manager.cc:933] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:41.522895 13843 tablet_copy_client.cc:323] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:41.525096 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet ae27dbfd0ab84c9d8c95896c0c4cf502 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:41.525667 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet ae27dbfd0ab84c9d8c95896c0c4cf502 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-ae27dbfd0ab84c9d8c95896c0c4cf502
I20260504 20:00:41.538151 12760 tablet_copy_source_session.cc:215] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:41.541399 13850 ts_tablet_manager.cc:933] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:41.541499 13843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae27dbfd0ab84c9d8c95896c0c4cf502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:41.543409 13850 tablet_copy_client.cc:323] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.2:41415
I20260504 20:00:41.545174 12835 tablet_copy_service.cc:140] P 5ff0ef72ec3b4667893a0420d3c7f923: Received BeginTabletCopySession request for tablet 7a80bf1f95a74ce09071a550fe15996f from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:48774)
I20260504 20:00:41.545742 12835 tablet_copy_service.cc:161] P 5ff0ef72ec3b4667893a0420d3c7f923: Beginning new tablet copy session on tablet 7a80bf1f95a74ce09071a550fe15996f from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:48774: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-7a80bf1f95a74ce09071a550fe15996f
I20260504 20:00:41.559397 12835 tablet_copy_source_session.cc:215] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:41.559938 13843 tablet_copy_client.cc:806] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:41.560672 13843 tablet_copy_client.cc:670] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:41.569991 13843 tablet_copy_client.cc:538] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:41.576716 13850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a80bf1f95a74ce09071a550fe15996f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:41.578879 13843 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:41.596755 13850 tablet_copy_client.cc:806] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:41.597424 13850 tablet_copy_client.cc:670] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:41.599925 13843 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:41.600697 13843 tablet_bootstrap.cc:492] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:41.601370 13843 ts_tablet_manager.cc:1403] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.002s
I20260504 20:00:41.605651 13850 tablet_copy_client.cc:538] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:41.606222 13843 raft_consensus.cc:359] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:41.607129 13843 raft_consensus.cc:740] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:41.607885 13843 consensus_queue.cc:260] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:41.610994 13843 ts_tablet_manager.cc:1434] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260504 20:00:41.612828 12760 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-ae27dbfd0ab84c9d8c95896c0c4cf502 received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:41.613261 12760 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-ae27dbfd0ab84c9d8c95896c0c4cf502 on tablet ae27dbfd0ab84c9d8c95896c0c4cf502 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:41.620986 13850 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:41.641197 13850 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:41.642320 13850 tablet_bootstrap.cc:492] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:41.643179 13850 ts_tablet_manager.cc:1403] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.004s
I20260504 20:00:41.646448 13850 raft_consensus.cc:359] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:41.647404 13850 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:41.648168 13850 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:41.652910 13850 ts_tablet_manager.cc:1434] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260504 20:00:41.654909 12835 tablet_copy_service.cc:342] P 5ff0ef72ec3b4667893a0420d3c7f923: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-7a80bf1f95a74ce09071a550fe15996f received from {username='slave'} at 127.0.0.1:48774
I20260504 20:00:41.655360 12835 tablet_copy_service.cc:434] P 5ff0ef72ec3b4667893a0420d3c7f923: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-7a80bf1f95a74ce09071a550fe15996f on tablet 7a80bf1f95a74ce09071a550fe15996f with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:41.767642 13531 raft_consensus.cc:1217] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:41.809618 13531 raft_consensus.cc:1217] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:42.044406 13531 raft_consensus.cc:1217] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:42.120585 13531 raft_consensus.cc:1217] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:42.286145 13775 raft_consensus.cc:1064] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:42.288885 13775 consensus_queue.cc:237] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:42.290838 13796 raft_consensus.cc:1064] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:42.293850 13796 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:42.299233 13531 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.300426 13856 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:42.305593 12896 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.308916 12896 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.309878 13533 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.311555 12817 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.313364 13832 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:42.315127 12742 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.315670 13834 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:42.316809 13796 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:42.320282 13775 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:42.330286 13775 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:42.356272 13832 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.359973 13811 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.362185 12742 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.372478 13532 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.374554 12896 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.394287 12892 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.435099 13531 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.437166 12646 catalog_manager.cc:5671] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.456867 12647 catalog_manager.cc:5671] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.462603 12818 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.497020 13832 raft_consensus.cc:1064] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:42.500073 13832 consensus_queue.cc:237] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:42.518466 13531 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.520000 12817 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.520284 12894 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.520767 13832 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:42.522919 13796 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:42.524977 13796 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:42.541258 12817 consensus_queue.cc:237] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:42.565574 13531 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:42.564751 13834 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.567589 13775 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:42.570048 12892 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.573129 13856 raft_consensus.cc:1064] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:42.574617 12742 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:42.575995 13856 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:42.582165 13811 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.584044 13531 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.593016 13532 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:42.594199 12818 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:42.595618 12892 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:42.598511 13532 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.599334 12648 catalog_manager.cc:5671] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.601233 13832 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:42.602253 13876 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:42.603323 13832 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:42.570083 12816 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.609588 12892 raft_consensus.cc:2955] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.611479 12896 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.616087 13533 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.616279 12742 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:42.623101 13856 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:42.624433 13876 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:42.624209 13866 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.626401 13533 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.616397 12645 catalog_manager.cc:5671] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.633723 13878 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:42.657460 12735 tablet_service.cc:1558] Processing DeleteTablet for tablet 57fe021572534bbdbbaea4aa4a64017a with delete_type TABLET_DATA_TOMBSTONED (TS 82787d8e64d24ca885ca517aa8ed1be2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57450
I20260504 20:00:42.664342 12633 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 57fe021572534bbdbbaea4aa4a64017a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:42.674050 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 410644ff9c0449188275a6d49ecc0c2d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:42.704792 12818 raft_consensus.cc:2955] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.713258 13881 tablet_replica.cc:333] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: stopping tablet replica
I20260504 20:00:42.744922 13881 raft_consensus.cc:2243] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:42.749084 13881 raft_consensus.cc:2272] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:42.876518 13881 ts_tablet_manager.cc:1916] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:42.887646 13811 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.891893 13532 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.906469 12645 catalog_manager.cc:5671] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.908806 13881 ts_tablet_manager.cc:1929] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:42.914055 12749 consensus_queue.cc:237] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:42.910384 13881 log.cc:1199] T 57fe021572534bbdbbaea4aa4a64017a P 82787d8e64d24ca885ca517aa8ed1be2: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/wals/57fe021572534bbdbbaea4aa4a64017a
I20260504 20:00:42.927820 12883 tablet_service.cc:1558] Processing DeleteTablet for tablet 410644ff9c0449188275a6d49ecc0c2d with delete_type TABLET_DATA_TOMBSTONED (TS 732c3560a688494196d22a1a48f89968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37306
I20260504 20:00:42.927997 13533 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:42.929716 12635 catalog_manager.cc:5002] TS 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): tablet 57fe021572534bbdbbaea4aa4a64017a (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:42.931780 13866 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:42.937728 12743 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.943920 12891 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:42.957057 12645 catalog_manager.cc:5671] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.972466 13796 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.975183 13533 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:42.985455 13832 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:42.989183 12818 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:42.998746 13532 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:43.000065 12743 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:43.001286 13856 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:43.004513 12647 catalog_manager.cc:5671] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:43.004812 12891 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:43.008855 13876 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:43.013572 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet ae27dbfd0ab84c9d8c95896c0c4cf502 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:43.013898 13878 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:43.015836 13532 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:43.032783 12633 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 7a80bf1f95a74ce09071a550fe15996f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:43.041530 12749 raft_consensus.cc:2955] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:43.046345 12648 catalog_manager.cc:5671] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:43.049100 12898 raft_consensus.cc:2955] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:43.071099 13885 tablet_replica.cc:333] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: stopping tablet replica
I20260504 20:00:43.087584 12883 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a80bf1f95a74ce09071a550fe15996f with delete_type TABLET_DATA_TOMBSTONED (TS 732c3560a688494196d22a1a48f89968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37306
I20260504 20:00:43.092691 12645 catalog_manager.cc:5218] ChangeConfig:REMOVE_PEER RPC for tablet ae27dbfd0ab84c9d8c95896c0c4cf502 with cas_config_opid_index 3: aborting the task: latest config opid_index 4; task opid_index 3
I20260504 20:00:43.098774 12806 tablet_service.cc:1558] Processing DeleteTablet for tablet ae27dbfd0ab84c9d8c95896c0c4cf502 with delete_type TABLET_DATA_TOMBSTONED (TS 5ff0ef72ec3b4667893a0420d3c7f923 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53094
I20260504 20:00:43.111048 13886 tablet_replica.cc:333] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: stopping tablet replica
I20260504 20:00:43.111927 13885 raft_consensus.cc:2243] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:43.112780 13886 raft_consensus.cc:2243] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:43.113912 13886 raft_consensus.cc:2272] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:43.120537 13886 ts_tablet_manager.cc:1916] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:43.126191 13885 raft_consensus.cc:2272] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:43.187822 13885 ts_tablet_manager.cc:1916] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:43.204981 13886 ts_tablet_manager.cc:1929] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:43.205832 13886 log.cc:1199] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/wals/ae27dbfd0ab84c9d8c95896c0c4cf502
I20260504 20:00:43.208305 12633 catalog_manager.cc:5002] TS 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): tablet ae27dbfd0ab84c9d8c95896c0c4cf502 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:43.216715 13885 ts_tablet_manager.cc:1929] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:43.217270 13885 log.cc:1199] T 410644ff9c0449188275a6d49ecc0c2d P 732c3560a688494196d22a1a48f89968: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/wals/410644ff9c0449188275a6d49ecc0c2d
I20260504 20:00:43.219327 12635 catalog_manager.cc:5002] TS 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): tablet 410644ff9c0449188275a6d49ecc0c2d (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:43.219539 13885 tablet_replica.cc:333] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: stopping tablet replica
I20260504 20:00:43.221019 13885 raft_consensus.cc:2243] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:43.222098 13885 raft_consensus.cc:2272] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:43.225615 13885 ts_tablet_manager.cc:1916] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:43.245951 13885 ts_tablet_manager.cc:1929] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:43.246585 13885 log.cc:1199] T 7a80bf1f95a74ce09071a550fe15996f P 732c3560a688494196d22a1a48f89968: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/wals/7a80bf1f95a74ce09071a550fe15996f
I20260504 20:00:43.248335 12635 catalog_manager.cc:5002] TS 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): tablet 7a80bf1f95a74ce09071a550fe15996f (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:44.410835 12896 consensus_queue.cc:237] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:44.417721 12743 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:44.417851 12816 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:44.418241 12864 consensus_peers.cc:597] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2114545cfec4b19a5a3d1d6751f73da. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:44.419368 13895 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:44.420446 13896 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:44.430572 13895 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.432392 12816 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.434674 12743 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.451651 12896 consensus_queue.cc:237] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:44.458508 12645 catalog_manager.cc:5671] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260504 20:00:44.470518 12864 consensus_peers.cc:597] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e44b757a449142a38ed04025e01cb4a5. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:44.472223 12816 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:44.474197 12743 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:44.474936 13895 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:44.476184 13896 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:44.487783 13896 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.495816 12816 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.498227 12743 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.518146 12896 consensus_queue.cc:237] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:44.538802 12646 catalog_manager.cc:5671] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.560757 12816 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:44.561429 12743 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:44.562146 12864 consensus_peers.cc:597] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f8ca9e82e464c93b50b298fc802e0d0. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:44.562425 13896 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:44.563997 13894 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:44.574266 13895 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.576114 12743 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.576186 12816 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.598557 12818 consensus_queue.cc:237] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:44.599308 12647 catalog_manager.cc:5671] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:44.631596 12743 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:44.633167 12896 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:44.634330 12790 consensus_peers.cc:597] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0832607bf7dd4d93a376269f0dd3dca1. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:44.635083 13897 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:44.636327 13892 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:44.650782 13892 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.653664 12896 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.655531 12743 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:44.679086 12647 catalog_manager.cc:5671] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
W20260504 20:00:44.807022 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800035040 after lost signal to thread 11539
W20260504 20:00:44.808184 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800091da0 after lost signal to thread 12682
W20260504 20:00:44.809494 12679 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee1c0 after lost signal to thread 12774
W20260504 20:00:44.810699 12679 debug-util.cc:398] Leaking SignalData structure 0x7b08002d78c0 after lost signal to thread 12849
W20260504 20:00:44.811877 12679 debug-util.cc:398] Leaking SignalData structure 0x7b08003197c0 after lost signal to thread 12923
W20260504 20:00:44.813102 12679 debug-util.cc:398] Leaking SignalData structure 0x7b080032d560 after lost signal to thread 13557
I20260504 20:00:44.915002 13915 ts_tablet_manager.cc:933] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:44.923394 13915 tablet_copy_client.cc:323] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.3:39151
I20260504 20:00:44.925282 12909 tablet_copy_service.cc:140] P 732c3560a688494196d22a1a48f89968: Received BeginTabletCopySession request for tablet f2114545cfec4b19a5a3d1d6751f73da from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:38968)
I20260504 20:00:44.925778 12909 tablet_copy_service.cc:161] P 732c3560a688494196d22a1a48f89968: Beginning new tablet copy session on tablet f2114545cfec4b19a5a3d1d6751f73da from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:38968: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-f2114545cfec4b19a5a3d1d6751f73da
I20260504 20:00:44.937686 12909 tablet_copy_source_session.cc:215] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:44.941455 13915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2114545cfec4b19a5a3d1d6751f73da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:44.957892 13915 tablet_copy_client.cc:806] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:44.958477 13915 tablet_copy_client.cc:670] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:44.965337 13915 tablet_copy_client.cc:538] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:44.975196 13915 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:44.978065 13917 ts_tablet_manager.cc:933] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:44.993898 13917 tablet_copy_client.cc:323] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.3:39151
I20260504 20:00:45.000643 12909 tablet_copy_service.cc:140] P 732c3560a688494196d22a1a48f89968: Received BeginTabletCopySession request for tablet e44b757a449142a38ed04025e01cb4a5 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:38968)
I20260504 20:00:45.001255 12909 tablet_copy_service.cc:161] P 732c3560a688494196d22a1a48f89968: Beginning new tablet copy session on tablet e44b757a449142a38ed04025e01cb4a5 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:38968: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-e44b757a449142a38ed04025e01cb4a5
I20260504 20:00:45.004329 13915 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:45.005429 13915 tablet_bootstrap.cc:492] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:45.006325 13915 ts_tablet_manager.cc:1403] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.001s
I20260504 20:00:45.010110 12909 tablet_copy_source_session.cc:215] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:45.013716 13917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44b757a449142a38ed04025e01cb4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:45.025310 13915 raft_consensus.cc:359] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:45.029466 13920 ts_tablet_manager.cc:933] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:45.026305 13915 raft_consensus.cc:740] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:45.034957 13915 consensus_queue.cc:260] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:45.037703 13917 tablet_copy_client.cc:806] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:45.043349 13917 tablet_copy_client.cc:670] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:45.044987 13920 tablet_copy_client.cc:323] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.3:39151
I20260504 20:00:45.047554 12908 tablet_copy_service.cc:140] P 732c3560a688494196d22a1a48f89968: Received BeginTabletCopySession request for tablet 0f8ca9e82e464c93b50b298fc802e0d0 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:38968)
I20260504 20:00:45.048154 12908 tablet_copy_service.cc:161] P 732c3560a688494196d22a1a48f89968: Beginning new tablet copy session on tablet 0f8ca9e82e464c93b50b298fc802e0d0 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:38968: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-0f8ca9e82e464c93b50b298fc802e0d0
I20260504 20:00:45.055539 13915 ts_tablet_manager.cc:1434] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.049s user 0.009s sys 0.004s
I20260504 20:00:45.057456 12908 tablet_copy_source_session.cc:215] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:45.060782 13917 tablet_copy_client.cc:538] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:45.062498 13920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f8ca9e82e464c93b50b298fc802e0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:45.073428 12908 tablet_copy_service.cc:342] P 732c3560a688494196d22a1a48f89968: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-f2114545cfec4b19a5a3d1d6751f73da received from {username='slave'} at 127.0.0.1:38968
I20260504 20:00:45.077564 12908 tablet_copy_service.cc:434] P 732c3560a688494196d22a1a48f89968: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-f2114545cfec4b19a5a3d1d6751f73da on tablet f2114545cfec4b19a5a3d1d6751f73da with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:45.087845 13917 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:45.287321 13920 tablet_copy_client.cc:806] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:45.288139 13920 tablet_copy_client.cc:670] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:45.298218 13915 ts_tablet_manager.cc:933] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:45.323637 13915 tablet_copy_client.cc:323] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.2:41415
I20260504 20:00:45.317544 13917 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:45.371282 13917 tablet_bootstrap.cc:492] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:45.374811 13917 ts_tablet_manager.cc:1403] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.287s user 0.047s sys 0.027s
I20260504 20:00:45.391413 12835 tablet_copy_service.cc:140] P 5ff0ef72ec3b4667893a0420d3c7f923: Received BeginTabletCopySession request for tablet 0832607bf7dd4d93a376269f0dd3dca1 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:48774)
I20260504 20:00:45.392503 12835 tablet_copy_service.cc:161] P 5ff0ef72ec3b4667893a0420d3c7f923: Beginning new tablet copy session on tablet 0832607bf7dd4d93a376269f0dd3dca1 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:48774: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-0832607bf7dd4d93a376269f0dd3dca1
I20260504 20:00:45.423835 13529 raft_consensus.cc:1217] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:45.426455 13920 tablet_copy_client.cc:538] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:45.431703 13917 raft_consensus.cc:359] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:45.432960 13917 raft_consensus.cc:740] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:45.474052 12835 tablet_copy_source_session.cc:215] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:45.476706 13920 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:45.454236 13917 consensus_queue.cc:260] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:45.491633 13917 ts_tablet_manager.cc:1434] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.116s user 0.009s sys 0.001s
I20260504 20:00:45.507984 12908 tablet_copy_service.cc:342] P 732c3560a688494196d22a1a48f89968: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-e44b757a449142a38ed04025e01cb4a5 received from {username='slave'} at 127.0.0.1:38968
I20260504 20:00:45.508455 12908 tablet_copy_service.cc:434] P 732c3560a688494196d22a1a48f89968: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-e44b757a449142a38ed04025e01cb4a5 on tablet e44b757a449142a38ed04025e01cb4a5 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:45.512157 13915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0832607bf7dd4d93a376269f0dd3dca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:45.552726 13920 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:45.555506 13920 tablet_bootstrap.cc:492] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:45.556794 13920 ts_tablet_manager.cc:1403] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.080s user 0.025s sys 0.018s
I20260504 20:00:45.563076 13920 raft_consensus.cc:359] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:45.565594 13920 raft_consensus.cc:740] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:45.566334 13920 consensus_queue.cc:260] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:45.570444 13920 ts_tablet_manager.cc:1434] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260504 20:00:45.572067 12908 tablet_copy_service.cc:342] P 732c3560a688494196d22a1a48f89968: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-0f8ca9e82e464c93b50b298fc802e0d0 received from {username='slave'} at 127.0.0.1:38968
I20260504 20:00:45.572538 12908 tablet_copy_service.cc:434] P 732c3560a688494196d22a1a48f89968: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-0f8ca9e82e464c93b50b298fc802e0d0 on tablet 0f8ca9e82e464c93b50b298fc802e0d0 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:45.585232 13915 tablet_copy_client.cc:806] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:45.585927 13915 tablet_copy_client.cc:670] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:45.590054 13915 tablet_copy_client.cc:538] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:45.602047 13915 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:45.640590 13915 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:45.641619 13915 tablet_bootstrap.cc:492] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:45.642409 13915 ts_tablet_manager.cc:1403] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.019s
I20260504 20:00:45.645643 13915 raft_consensus.cc:359] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:45.646605 13915 raft_consensus.cc:740] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:45.647317 13915 consensus_queue.cc:260] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:45.650128 13915 ts_tablet_manager.cc:1434] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260504 20:00:45.652463 12835 tablet_copy_service.cc:342] P 5ff0ef72ec3b4667893a0420d3c7f923: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-0832607bf7dd4d93a376269f0dd3dca1 received from {username='slave'} at 127.0.0.1:48774
I20260504 20:00:45.652948 12835 tablet_copy_service.cc:434] P 5ff0ef72ec3b4667893a0420d3c7f923: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-0832607bf7dd4d93a376269f0dd3dca1 on tablet 0832607bf7dd4d93a376269f0dd3dca1 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:45.687119 13531 raft_consensus.cc:1217] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:45.815918 13531 raft_consensus.cc:1217] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:45.959673 13941 raft_consensus.cc:1064] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:45.962754 13941 consensus_queue.cc:237] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:45.978318 13531 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:45.978344 12825 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:45.980391 13936 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:45.981792 13933 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:46.020721 12742 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.064644 13531 raft_consensus.cc:1217] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260504 20:00:46.089890 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 50308.7us
I20260504 20:00:46.095680 13896 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:46.105648 13896 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.138062 12825 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.147629 13531 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.162079 12647 catalog_manager.cc:5671] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:46.169296 13892 raft_consensus.cc:1064] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:46.174589 13892 consensus_queue.cc:237] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:46.179543 12742 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.222726 13531 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.223977 12892 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.225715 12748 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.225595 13925 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:46.227846 13924 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:46.230837 13925 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:46.296020 13892 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.302313 12892 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.322558 12748 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.372844 12892 consensus_queue.cc:237] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:46.383034 13533 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.404613 12648 catalog_manager.cc:5671] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.419091 13533 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:46.427098 13933 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:46.431886 13895 raft_consensus.cc:1064] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:46.436851 13895 consensus_queue.cc:237] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:46.449717 12825 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:46.450425 13896 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.455247 13936 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:46.463428 12817 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.469071 12816 consensus_queue.cc:237] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:46.472460 13533 raft_consensus.cc:2955] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.475903 13531 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:46.477461 12748 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:46.481005 13957 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:46.490152 13957 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:00:46.494642 12742 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.492882 13957 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.501051 12748 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.504017 13936 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:46.505883 13529 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.507592 12825 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.508967 13531 raft_consensus.cc:2955] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.525883 13895 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:46.532671 13933 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:46.533782 12633 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 0832607bf7dd4d93a376269f0dd3dca1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:46.552357 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet f2114545cfec4b19a5a3d1d6751f73da with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:46.581411 12648 catalog_manager.cc:5671] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 3 to 4, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:00:46.585968 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 58166.4us
I20260504 20:00:46.592757 12647 catalog_manager.cc:5671] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:46.606428 13933 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.624490 12735 tablet_service.cc:1558] Processing DeleteTablet for tablet f2114545cfec4b19a5a3d1d6751f73da with delete_type TABLET_DATA_TOMBSTONED (TS 82787d8e64d24ca885ca517aa8ed1be2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57450
I20260504 20:00:46.625377 12883 tablet_service.cc:1558] Processing DeleteTablet for tablet 0832607bf7dd4d93a376269f0dd3dca1 with delete_type TABLET_DATA_TOMBSTONED (TS 732c3560a688494196d22a1a48f89968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37306
I20260504 20:00:46.644057 13533 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.669224 13962 raft_consensus.cc:1064] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:46.706413 13962 consensus_queue.cc:237] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:46.725006 13969 tablet_replica.cc:333] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: stopping tablet replica
I20260504 20:00:46.729260 13968 tablet_replica.cc:333] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: stopping tablet replica
I20260504 20:00:46.730437 13969 raft_consensus.cc:2243] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:46.894178 12742 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.894198 12743 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.900843 13943 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:46.942332 13969 raft_consensus.cc:2272] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:46.949417 12818 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:46.953301 13969 ts_tablet_manager.cc:1916] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:46.959182 13968 raft_consensus.cc:2243] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:46.960216 13968 raft_consensus.cc:2272] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:46.972715 12818 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:46.974313 13962 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:46.979720 13969 ts_tablet_manager.cc:1929] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:46.980583 13969 log.cc:1199] T 0832607bf7dd4d93a376269f0dd3dca1 P 732c3560a688494196d22a1a48f89968: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/wals/0832607bf7dd4d93a376269f0dd3dca1
I20260504 20:00:46.982933 13968 ts_tablet_manager.cc:1916] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:46.991382 12645 catalog_manager.cc:5671] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.009152 13530 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:47.010488 12635 catalog_manager.cc:5002] TS 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): tablet 0832607bf7dd4d93a376269f0dd3dca1 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:47.013130 13936 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:47.072693 13968 ts_tablet_manager.cc:1929] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:47.073282 13968 log.cc:1199] T f2114545cfec4b19a5a3d1d6751f73da P 82787d8e64d24ca885ca517aa8ed1be2: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/wals/f2114545cfec4b19a5a3d1d6751f73da
I20260504 20:00:47.072312 13895 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.074108 12818 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.080624 12635 catalog_manager.cc:5002] TS 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): tablet f2114545cfec4b19a5a3d1d6751f73da (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:47.136494 13533 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.116497 12893 consensus_queue.cc:237] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:47.149271 12742 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.170871 13533 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:47.172847 13962 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:47.201843 12645 catalog_manager.cc:5671] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:47.205513 12749 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:47.202101 13896 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.209352 13533 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.229080 13933 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:47.236905 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet e44b757a449142a38ed04025e01cb4a5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:47.249904 12898 consensus_queue.cc:237] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:47.259740 13533 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:47.263906 13943 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:47.269963 12749 raft_consensus.cc:2955] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.274030 13963 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.282568 13533 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.319712 12744 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:47.339990 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 0f8ca9e82e464c93b50b298fc802e0d0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:47.345654 12645 catalog_manager.cc:5671] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:47.366389 12806 tablet_service.cc:1558] Processing DeleteTablet for tablet e44b757a449142a38ed04025e01cb4a5 with delete_type TABLET_DATA_TOMBSTONED (TS 5ff0ef72ec3b4667893a0420d3c7f923 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53094
I20260504 20:00:47.369858 13943 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:47.379568 13978 tablet_replica.cc:333] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: stopping tablet replica
I20260504 20:00:47.381376 13978 raft_consensus.cc:2243] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:47.394124 13978 raft_consensus.cc:2272] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:47.398741 13978 ts_tablet_manager.cc:1916] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:47.376840 12647 catalog_manager.cc:5671] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New cstate: current_term: 1 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.423697 12749 raft_consensus.cc:2955] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:47.425150 13978 ts_tablet_manager.cc:1929] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:47.425890 13978 log.cc:1199] T e44b757a449142a38ed04025e01cb4a5 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/wals/e44b757a449142a38ed04025e01cb4a5
I20260504 20:00:47.428654 12633 catalog_manager.cc:5002] TS 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): tablet e44b757a449142a38ed04025e01cb4a5 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:00:47.436331 12806 tablet_service.cc:1558] Processing DeleteTablet for tablet 0f8ca9e82e464c93b50b298fc802e0d0 with delete_type TABLET_DATA_TOMBSTONED (TS 5ff0ef72ec3b4667893a0420d3c7f923 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53094
I20260504 20:00:47.437762 13978 tablet_replica.cc:333] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: stopping tablet replica
I20260504 20:00:47.439607 13978 raft_consensus.cc:2243] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:00:47.440675 13978 raft_consensus.cc:2272] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:00:47.451937 13978 ts_tablet_manager.cc:1916] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:00:47.488323 13978 ts_tablet_manager.cc:1929] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:00:47.495898 13978 log.cc:1199] T 0f8ca9e82e464c93b50b298fc802e0d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/wals/0f8ca9e82e464c93b50b298fc802e0d0
I20260504 20:00:47.497830 12633 catalog_manager.cc:5002] TS 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): tablet 0f8ca9e82e464c93b50b298fc802e0d0 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
W20260504 20:00:47.785727 12846 debug-util.cc:398] Leaking SignalData structure 0x7b080032f580 after lost signal to thread 11539
W20260504 20:00:47.786847 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002e8040 after lost signal to thread 12682
W20260504 20:00:47.787869 12846 debug-util.cc:398] Leaking SignalData structure 0x7b0800276260 after lost signal to thread 12774
W20260504 20:00:47.789036 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002d5460 after lost signal to thread 12849
W20260504 20:00:47.790046 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08002ed8c0 after lost signal to thread 12923
W20260504 20:00:47.790977 12846 debug-util.cc:398] Leaking SignalData structure 0x7b08001e6de0 after lost signal to thread 13557
W20260504 20:00:48.531502 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 58593.5us
I20260504 20:00:48.531105 12898 consensus_queue.cc:237] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
W20260504 20:00:48.559942 12864 consensus_peers.cc:597] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 593a993b4ff341c9b98629e2c051ce00. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:48.561597 12749 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:48.563403 13963 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:48.566810 12825 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:48.572717 13895 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:48.666774 13943 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:48.696623 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:48.742775 12825 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
W20260504 20:00:48.858269 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 74729.4us
I20260504 20:00:48.865983 12742 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
W20260504 20:00:48.872558 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 98237.5us
I20260504 20:00:48.894236 12647 catalog_manager.cc:5671] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
W20260504 20:00:48.910593 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 50995.8us
I20260504 20:00:48.911378 12744 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
W20260504 20:00:48.929102 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 54861.9us
I20260504 20:00:48.936650 12825 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:48.937665 12825 raft_consensus.cc:606] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:48.940675 12817 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47a6e494e974412e9a7112344021d967"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:48.941509 12817 raft_consensus.cc:606] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
W20260504 20:00:48.959014 12715 consensus_peers.cc:597] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91ff767ed7c24b97940a459421881055. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:48.960035 12825 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:48.962219 13973 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:48.995774 12898 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:48.999734 13973 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:49.012441 13984 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.016650 12744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:00:49.017307 12744 raft_consensus.cc:606] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:49.026218 12818 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:49.027098 12818 raft_consensus.cc:606] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:49.043393 13973 raft_consensus.cc:993] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: : Instructing follower 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:49.044317 13973 raft_consensus.cc:1081] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Signalling peer 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:49.047158 12818 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:53098
I20260504 20:00:49.049283 12818 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:49.049649 13988 raft_consensus.cc:993] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:49.049983 12818 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.050302 13979 raft_consensus.cc:1081] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:49.052139 13528 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40924
I20260504 20:00:49.052752 13528 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:49.053139 13528 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.059289 12744 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:49.061127 12896 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.064733 12818 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.067703 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:49.068372 12749 raft_consensus.cc:3055] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:49.068655 12749 raft_consensus.cc:740] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:49.072171 12749 consensus_queue.cc:260] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.073537 12749 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.075142 13528 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.070960 12825 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.078691 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:49.079299 12742 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.087141 12742 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:49.088945 13499 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:49.091152 12898 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:49.092439 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:49.093066 12817 raft_consensus.cc:3055] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:49.092991 13982 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:49.093531 12817 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:49.094331 12817 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.095467 12817 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.102598 12818 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:49.102784 12715 consensus_peers.cc:597] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfa79aedaea6499facdfc14a3e7467f8. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:49.106261 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:49.106964 12893 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.107187 12816 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:49.108722 13984 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:49.124523 14011 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:49.135088 13983 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.135187 12645 catalog_manager.cc:5671] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.143330 12816 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.091727 13528 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:49.148363 12896 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:00:49.156051 12896 raft_consensus.cc:606] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:49.152607 12893 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:49.161442 12817 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:49.161458 12821 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:49.162343 12821 raft_consensus.cc:606] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:49.163475 14014 ts_tablet_manager.cc:933] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:49.164260 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260504 20:00:49.166123 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:49.171690 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:49.161552 12791 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:49.180722 13988 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:49.181634 13988 raft_consensus.cc:697] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:49.154366 12749 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:49.182823 13988 consensus_queue.cc:237] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.215826 14014 tablet_copy_client.cc:323] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.3:39151
I20260504 20:00:49.219105 12898 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.230177 12908 tablet_copy_service.cc:140] P 732c3560a688494196d22a1a48f89968: Received BeginTabletCopySession request for tablet 593a993b4ff341c9b98629e2c051ce00 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:38968)
I20260504 20:00:49.231087 12908 tablet_copy_service.cc:161] P 732c3560a688494196d22a1a48f89968: Beginning new tablet copy session on tablet 593a993b4ff341c9b98629e2c051ce00 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:38968: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-593a993b4ff341c9b98629e2c051ce00
I20260504 20:00:49.265446 12908 tablet_copy_source_session.cc:215] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:49.271941 14011 raft_consensus.cc:697] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:49.275372 14014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593a993b4ff341c9b98629e2c051ce00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:49.272809 12647 catalog_manager.cc:5671] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.278740 12816 consensus_queue.cc:237] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
W20260504 20:00:49.280927 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 61764.3us
I20260504 20:00:49.274976 14011 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.314636 14000 raft_consensus.cc:993] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:00:49.315164 14000 raft_consensus.cc:1081] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:00:49.316202 12742 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 20:00:49.316494 12790 consensus_peers.cc:597] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 -> Peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Couldn't send request to peer ef6057d9a49e4942b47ea0fcd66ab6f7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e8eba4dbb8d436f81062ca3bbf46788. This is attempt 1: this message will repeat every 5th retry.
I20260504 20:00:49.319022 12893 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 20:00:49.319501 12896 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47a6e494e974412e9a7112344021d967"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37318
I20260504 20:00:49.320076 12896 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:49.319947 13979 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:49.320494 12896 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.321950 13979 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:49.330605 13997 raft_consensus.cc:993] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: : Instructing follower 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:49.330425 12896 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.346722 13997 raft_consensus.cc:1081] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Signalling peer 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:49.349944 12816 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:53102
I20260504 20:00:49.350556 12816 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:49.350924 12816 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.351572 12648 catalog_manager.cc:5671] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:49.352620 12647 catalog_manager.cc:5671] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:49.354350 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:49.354954 12825 raft_consensus.cc:3055] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:49.355253 12825 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:49.344146 14014 tablet_copy_client.cc:806] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:49.356022 12825 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.356490 14014 tablet_copy_client.cc:670] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:49.358695 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:49.359337 12742 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.359763 12825 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.366117 14000 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.368148 12896 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:49.369194 12816 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.371320 12816 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:49.373589 12744 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.376314 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:49.376924 12749 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.379143 12825 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:49.380353 12863 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:49.381248 13997 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:49.381742 13997 raft_consensus.cc:697] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:49.382704 13997 consensus_queue.cc:237] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.391842 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:49.392515 12893 raft_consensus.cc:3055] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:49.392843 12893 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:49.393669 12893 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.395084 12893 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.406522 13979 raft_consensus.cc:993] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:49.407176 13979 raft_consensus.cc:1081] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:49.408488 12893 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:49.410858 13533 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40924
I20260504 20:00:49.413160 13533 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:49.413573 13533 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.422797 13533 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.373617 14014 tablet_copy_client.cc:538] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:49.433419 13533 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:49.434190 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:49.434867 12816 raft_consensus.cc:3055] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:49.434883 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:49.435259 12816 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:49.435671 12745 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.436148 12816 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.438515 12816 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.442122 12791 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:49.443490 13960 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:49.444065 13960 raft_consensus.cc:697] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:49.445020 13960 consensus_queue.cc:237] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:49.453442 12742 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:49.454438 12745 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:49.457942 13499 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:49.459071 14011 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:49.459599 14011 raft_consensus.cc:697] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:49.460580 14011 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.417502 12749 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:49.477794 12648 catalog_manager.cc:5671] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.472659 12816 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:49.494306 12647 catalog_manager.cc:5671] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:49.512307 12645 catalog_manager.cc:5671] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:49.518486 12646 catalog_manager.cc:5671] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:49.546686 14014 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
W20260504 20:00:49.551142 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 60311.1us
I20260504 20:00:49.563637 12898 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } } }
I20260504 20:00:49.636866 14030 ts_tablet_manager.cc:933] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:49.639114 14000 raft_consensus.cc:993] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:49.648012 14000 raft_consensus.cc:1081] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:49.650156 13533 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40924
I20260504 20:00:49.650815 13533 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:49.651160 13533 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.665527 13533 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.668321 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:49.668941 12744 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.678020 12744 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:49.684007 13499 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:49.684927 14011 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:49.685484 14011 raft_consensus.cc:697] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:49.686475 14011 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.686957 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:49.687615 12816 raft_consensus.cc:3055] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:49.687925 12816 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:49.696599 12816 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:49.698653 12816 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:49.723531 14030 tablet_copy_client.cc:323] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:49.725270 12896 raft_consensus.cc:1275] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:49.728307 12647 catalog_manager.cc:5671] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:49.728477 12760 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet 91ff767ed7c24b97940a459421881055 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:49.732884 13979 consensus_queue.cc:1048] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:49.733949 12760 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet 91ff767ed7c24b97940a459421881055 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-91ff767ed7c24b97940a459421881055
I20260504 20:00:49.734077 13533 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:49.737756 12816 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:49.755560 14014 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:49.756819 14014 tablet_bootstrap.cc:492] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:49.757735 14014 ts_tablet_manager.cc:1403] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.211s user 0.023s sys 0.007s
I20260504 20:00:49.761338 14014 raft_consensus.cc:359] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:49.763335 14014 raft_consensus.cc:740] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:49.764170 14014 consensus_queue.cc:260] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:49.767520 14014 ts_tablet_manager.cc:1434] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260504 20:00:49.769464 12908 tablet_copy_service.cc:342] P 732c3560a688494196d22a1a48f89968: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-593a993b4ff341c9b98629e2c051ce00 received from {username='slave'} at 127.0.0.1:38968
I20260504 20:00:49.770263 12908 tablet_copy_service.cc:434] P 732c3560a688494196d22a1a48f89968: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-593a993b4ff341c9b98629e2c051ce00 on tablet 593a993b4ff341c9b98629e2c051ce00 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:49.733565 14034 ts_tablet_manager.cc:933] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:49.786324 14034 tablet_copy_client.cc:323] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.1:44007
I20260504 20:00:49.788686 12759 tablet_copy_service.cc:140] P 82787d8e64d24ca885ca517aa8ed1be2: Received BeginTabletCopySession request for tablet dfa79aedaea6499facdfc14a3e7467f8 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:57934)
I20260504 20:00:49.789319 12759 tablet_copy_service.cc:161] P 82787d8e64d24ca885ca517aa8ed1be2: Beginning new tablet copy session on tablet dfa79aedaea6499facdfc14a3e7467f8 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:57934: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-dfa79aedaea6499facdfc14a3e7467f8
I20260504 20:00:49.799409 14014 ts_tablet_manager.cc:933] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:49.806121 12760 tablet_copy_source_session.cc:215] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:49.808678 12759 tablet_copy_source_session.cc:215] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:49.813609 14034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa79aedaea6499facdfc14a3e7467f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:49.821179 12749 raft_consensus.cc:1275] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:49.831281 14030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ff767ed7c24b97940a459421881055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:49.836011 12745 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:49.836225 12744 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:49.844282 14029 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:49.864176 14016 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:49.865501 14014 tablet_copy_client.cc:323] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.68.2:41415
I20260504 20:00:49.868826 14000 consensus_queue.cc:1048] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:49.901141 14030 tablet_copy_client.cc:806] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:49.901995 14030 tablet_copy_client.cc:670] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:49.910553 12835 tablet_copy_service.cc:140] P 5ff0ef72ec3b4667893a0420d3c7f923: Received BeginTabletCopySession request for tablet 0e8eba4dbb8d436f81062ca3bbf46788 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 ({username='slave'} at 127.0.0.1:48774)
I20260504 20:00:49.910889 12816 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:49.911204 12835 tablet_copy_service.cc:161] P 5ff0ef72ec3b4667893a0420d3c7f923: Beginning new tablet copy session on tablet 0e8eba4dbb8d436f81062ca3bbf46788 from peer ef6057d9a49e4942b47ea0fcd66ab6f7 at {username='slave'} at 127.0.0.1:48774: session id = ef6057d9a49e4942b47ea0fcd66ab6f7-0e8eba4dbb8d436f81062ca3bbf46788
I20260504 20:00:49.924353 14011 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:49.922646 14034 tablet_copy_client.cc:806] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:49.926205 14034 tablet_copy_client.cc:670] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
I20260504 20:00:49.932435 14030 tablet_copy_client.cc:538] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:49.933660 14034 tablet_copy_client.cc:538] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:49.952277 12835 tablet_copy_source_session.cc:215] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Tablet Copy: opened 0 blocks and 1 log segments
I20260504 20:00:49.960726 14014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8eba4dbb8d436f81062ca3bbf46788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 20:00:49.968788 12816 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:49.976994 12818 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:49.984890 14029 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:49.978222 14011 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260504 20:00:50.002983 12748 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:50.009429 12893 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:50.010805 14000 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260504 20:00:50.018270 13979 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260504 20:00:50.148000 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 100258us
I20260504 20:00:50.215483 14030 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:50.258949 14034 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:50.266122 14014 tablet_copy_client.cc:806] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 0 data blocks...
I20260504 20:00:50.266916 14014 tablet_copy_client.cc:670] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Starting download of 1 WAL segments...
W20260504 20:00:50.285190 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 163265us
I20260504 20:00:50.342492 12816 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:50.356870 13533 raft_consensus.cc:1217] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260504 20:00:50.381893 12744 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:50.398824 12745 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:50.401824 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:50.523434 12744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:00:50.524124 12744 raft_consensus.cc:606] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:00:50.525415 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 179891us
W20260504 20:00:50.526960 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 125855us
I20260504 20:00:50.540138 14030 tablet_bootstrap.cc:700] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent opening tablet: real 0.103s user 0.001s sys 0.000s
I20260504 20:00:50.469949 14011 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260504 20:00:50.551136 14011 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:00:50.575016 12821 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:50.575770 12821 raft_consensus.cc:606] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:50.578191 14067 raft_consensus.cc:993] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:50.578761 14067 raft_consensus.cc:1081] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:50.602362 14076 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:50.611392 13533 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40916
I20260504 20:00:50.612160 13533 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:50.612594 13533 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.627430 12896 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:00:50.628233 12896 raft_consensus.cc:606] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:50.640658 14030 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260504 20:00:50.643667 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 44929.3us
I20260504 20:00:50.652876 14030 tablet_bootstrap.cc:492] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:50.655032 12896 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:00:50.662775 12896 raft_consensus.cc:606] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:50.664330 14030 ts_tablet_manager.cc:1403] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.449s user 0.019s sys 0.029s
I20260504 20:00:50.680394 12816 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:50.681097 12816 raft_consensus.cc:606] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:00:50.672120 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 188875us
W20260504 20:00:50.666713 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 90594.9us
I20260504 20:00:50.701751 14014 tablet_copy_client.cc:538] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260504 20:00:50.702134 14070 raft_consensus.cc:993] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: : Instructing follower 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:50.703094 14070 raft_consensus.cc:1081] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Signalling peer 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:50.704979 12825 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:53102
I20260504 20:00:50.705552 12825 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:50.705953 12825 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.706144 12896 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:00:50.706890 12896 raft_consensus.cc:606] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:50.708133 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260504 20:00:50.710060 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:50.710855 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:50.737666 12825 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:50.742395 14060 raft_consensus.cc:993] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:50.742904 14060 raft_consensus.cc:1081] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:50.680079 14030 raft_consensus.cc:359] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:50.743422 14030 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:50.747627 14014 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap starting.
I20260504 20:00:50.748715 14030 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:50.759229 14030 ts_tablet_manager.cc:1434] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.082s user 0.006s sys 0.001s
I20260504 20:00:50.741947 13533 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:50.766485 12825 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:50.761746 13533 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:50.769125 13528 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40940
I20260504 20:00:50.777969 13528 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:50.778393 13528 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.772472 12760 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-91ff767ed7c24b97940a459421881055 received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:50.781667 12760 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-91ff767ed7c24b97940a459421881055 on tablet 91ff767ed7c24b97940a459421881055 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:50.773435 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:50.788038 12749 raft_consensus.cc:3055] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:50.788523 12749 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:50.789678 13527 raft_consensus.cc:1217] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:50.794118 12749 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:50.800139 12749 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.797555 14034 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:50.815192 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:50.816145 12825 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.821091 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:50.820056 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:50.825389 12893 raft_consensus.cc:3055] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:50.825759 12893 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:50.826687 12744 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.829795 12893 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:50.832617 12893 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.863186 12744 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:50.870366 12791 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:50.876329 12893 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:50.879810 13528 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:50.905444 14058 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:50.906322 14058 raft_consensus.cc:697] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:50.913087 13528 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:50.902063 12749 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:50.918035 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:50.902100 14034 tablet_bootstrap.cc:492] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:50.918560 12893 raft_consensus.cc:3055] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:50.919572 12893 raft_consensus.cc:740] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:50.924794 14034 ts_tablet_manager.cc:1403] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.672s user 0.039s sys 0.021s
I20260504 20:00:50.926103 12893 consensus_queue.cc:260] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:50.927460 12893 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.929936 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:50.930974 12816 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:50.931316 12825 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:50.935369 13497 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:50.907672 14058 consensus_queue.cc:237] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:50.950107 14060 raft_consensus.cc:1064] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:50.955992 12893 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:50.953681 12816 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
W20260504 20:00:50.957127 14084 log.cc:927] Time spent T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20260504 20:00:50.953526 14085 log.cc:927] Time spent T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260504 20:00:50.962070 14077 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:50.962867 14077 raft_consensus.cc:697] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:50.994448 13499 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:50.994905 14086 raft_consensus.cc:993] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:50.996099 14076 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:51.004457 13925 raft_consensus.cc:1081] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:50.963985 14034 raft_consensus.cc:359] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:50.993314 14060 consensus_queue.cc:237] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:50.998571 14076 raft_consensus.cc:697] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:51.015287 13527 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40924
I20260504 20:00:51.024120 13527 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:50.979012 14077 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.012729 14034 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
I20260504 20:00:51.036733 14034 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:51.032088 14076 consensus_queue.cc:237] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.024742 13527 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:51.057967 13988 raft_consensus.cc:993] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:51.058696 13988 raft_consensus.cc:1081] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:51.066898 13533 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40924
I20260504 20:00:51.067698 13533 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:51.068151 13533 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:51.072335 13527 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.194015 14034 ts_tablet_manager.cc:1434] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.269s user 0.017s sys 0.002s
I20260504 20:00:51.194103 13527 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:51.198855 13997 raft_consensus.cc:993] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: : Instructing follower 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:51.204595 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:51.205353 12825 raft_consensus.cc:3055] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:51.205690 12825 raft_consensus.cc:740] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:51.206530 12825 consensus_queue.cc:260] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.213016 12825 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:00:51.213068 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 137781us
I20260504 20:00:51.220811 12646 catalog_manager.cc:5671] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:00:51.225116 13533 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.238468 13533 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:51.248646 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:51.249524 12817 raft_consensus.cc:3055] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:51.249935 12817 raft_consensus.cc:740] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:51.250945 12817 consensus_queue.cc:260] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.252485 12817 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:51.266319 12760 tablet_copy_service.cc:342] P 82787d8e64d24ca885ca517aa8ed1be2: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-dfa79aedaea6499facdfc14a3e7467f8 received from {username='slave'} at 127.0.0.1:57934
I20260504 20:00:51.267180 12760 tablet_copy_service.cc:434] P 82787d8e64d24ca885ca517aa8ed1be2: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-dfa79aedaea6499facdfc14a3e7467f8 on tablet dfa79aedaea6499facdfc14a3e7467f8 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:51.296887 14088 raft_consensus.cc:1081] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Signalling peer 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:51.339146 12817 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:51.354574 12825 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:51.362025 12743 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260504 20:00:51.381011 13497 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:51.387077 13497 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:51.395571 12821 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:53102
I20260504 20:00:51.396212 12821 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:51.396545 12821 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:51.399456 12825 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260504 20:00:51.401846 13533 raft_consensus.cc:1217] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:51.408977 13533 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEARNER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260504 20:00:51.389787 14078 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:51.455379 12647 catalog_manager.cc:5671] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:51.437698 14078 raft_consensus.cc:697] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:51.430064 14088 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20260504 20:00:51.470544 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 183448us
I20260504 20:00:51.471109 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:51.472208 12892 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:51.474272 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:51.478325 12894 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:51.488178 14074 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:51.489152 14074 raft_consensus.cc:697] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:51.490583 14074 consensus_queue.cc:237] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.505125 12749 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:51.501101 14078 consensus_queue.cc:237] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.518818 12892 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:51.483824 13997 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260504 20:00:51.503819 12821 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:51.548091 14014 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 20:00:51.514127 14099 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260504 20:00:51.559147 12894 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:51.564452 12893 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:51.564725 12821 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:51.572642 14076 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:00:51.573459 14103 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260504 20:00:51.576781 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:51.577437 12744 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:51.598423 12821 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260504 20:00:51.601779 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 78976.6us
I20260504 20:00:51.609037 14077 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:51.619300 14014 tablet_bootstrap.cc:492] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Bootstrap complete.
I20260504 20:00:51.620328 14014 ts_tablet_manager.cc:1403] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent bootstrapping tablet: real 0.897s user 0.024s sys 0.029s
I20260504 20:00:51.620579 12749 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:51.622803 14051 raft_consensus.cc:1064] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:51.626276 12744 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
W20260504 20:00:51.628567 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 44963.4us
I20260504 20:00:51.631407 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:51.635360 12896 raft_consensus.cc:3055] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:51.635720 12896 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:51.635114 12749 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:51.638136 12791 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:51.643482 14077 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260504 20:00:51.655290 13979 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:51.664006 12896 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:51.667621 12894 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:51.658449 13979 raft_consensus.cc:697] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:51.644300 14051 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:51.736222 14087 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260504 20:00:51.681375 14093 consensus_queue.cc:1059] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260504 20:00:51.757438 12896 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:00:51.762279 14106 log.cc:927] Time spent T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260504 20:00:51.781868 13979 consensus_queue.cc:237] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:51.789976 12645 catalog_manager.cc:5671] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:51.822616 12897 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:51.819955 12896 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:51.848311 13533 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260504 20:00:51.866395 14109 log.cc:927] Time spent T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.070s user 0.002s sys 0.000s
W20260504 20:00:51.893698 14114 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.078s user 0.000s sys 0.005s
I20260504 20:00:51.926080 12819 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:51.901999 14048 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
W20260504 20:00:52.019500 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:52.024906 13974 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260504 20:00:52.028512 14014 raft_consensus.cc:359] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
W20260504 20:00:52.044987 14108 log.cc:927] Time spent T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260504 20:00:52.047807 14051 raft_consensus.cc:1064] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:52.029788 14014 raft_consensus.cc:740] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Initialized, Role: LEARNER
W20260504 20:00:52.056730 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 181437us
I20260504 20:00:52.054093 12892 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260504 20:00:52.055231 14110 log.cc:927] Time spent T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32: Append to log took a long time: real 0.105s user 0.000s sys 0.004s
I20260504 20:00:52.083938 12816 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:52.125823 12821 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:52.142043 12821 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:52.115062 14076 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260504 20:00:52.100344 14014 consensus_queue.cc:260] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:52.166369 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260504 20:00:52.200354 12744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.023s sys 0.022s
W20260504 20:00:52.210007 12744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s user 0.024s sys 0.022s
I20260504 20:00:52.222396 12818 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:52.188371 14112 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
W20260504 20:00:52.250095 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 364792us
W20260504 20:00:52.268193 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 92642.1us
W20260504 20:00:52.364081 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 92246.3us
I20260504 20:00:52.370044 14076 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260504 20:00:52.371647 14122 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
W20260504 20:00:52.382606 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:52.351774 14112 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260504 20:00:52.394017 14014 ts_tablet_manager.cc:1434] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent starting tablet: real 0.773s user 0.023s sys 0.003s
W20260504 20:00:52.421185 14127 log.cc:927] Time spent T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
W20260504 20:00:52.423040 14128 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260504 20:00:52.426527 12744 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:52.428085 14051 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:00:52.415376 14014 ts_tablet_manager.cc:1467] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Tablet startup took 794ms
I20260504 20:00:52.473268 12896 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:00:52.474191 12896 raft_consensus.cc:606] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:00:52.478988 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:52.496384 13533 raft_consensus.cc:1217] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260504 20:00:52.503937 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:52.520473 12819 raft_consensus.cc:606] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:00:52.546082 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:52.550647 13533 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:52.413556 14094 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:52.586763 14063 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260504 20:00:52.597903 12895 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:52.610431 14065 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20260504 20:00:52.612880 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 425635us
I20260504 20:00:52.542034 14132 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:52.597939 13988 raft_consensus.cc:993] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:52.632565 13988 raft_consensus.cc:1081] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
W20260504 20:00:52.534551 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 264833us
I20260504 20:00:52.636973 13533 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40924
I20260504 20:00:52.641861 13533 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:52.642422 13533 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:00:52.464394 14014 ts_tablet_manager.cc:1469] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Trace:
0504 20:00:49.741116 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:49.741292 (+ 176us) service_pool.cc:224] Handling call
0504 20:00:49.799874 (+ 58582us) ts_tablet_manager.cc:934] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Initiating tablet copy from peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
0504 20:00:49.959470 (+159596us) tablet_copy_client.cc:368] Tablet copy session begun
0504 20:00:50.051894 (+ 92424us) tablet_metadata.cc:665] Metadata flushed
0504 20:00:50.052090 (+ 196us) tablet_copy_client.cc:497] Wrote new tablet metadata
0504 20:00:50.264882 (+212792us) tablet_copy_client.cc:418] Wrote new consensus metadata
0504 20:00:50.265232 (+ 350us) tablet_replica.cc:206] Creating consensus instance
0504 20:00:50.265577 (+ 345us) ts_tablet_manager.cc:989] Replica registered.
0504 20:00:50.265638 (+ 61us) inbound_call.cc:177] Queueing success response
0504 20:00:50.723240 (+457602us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0504 20:00:51.620716 (+897476us) ts_tablet_manager.cc:1435] Starting tablet replica
0504 20:00:51.621352 (+ 636us) tablet_replica.cc:261] Starting instrumentation
0504 20:00:51.656798 (+ 35446us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":287,"raft.queue_time_us":7512,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":6289,"tablet-copy.queue_time_us":57502,"tablet-download-pool.queue_time_us":122,"tablet-download-pool.run_cpu_time_us":1451,"tablet-download-pool.run_wall_time_us":432847,"thread_start_us":133624,"threads_started":2}
I20260504 20:00:52.710841 12897 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:52.733556 13533 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:52.646342 14132 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:52.770197 13533 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:52.789716 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 78398.1us
W20260504 20:00:52.802002 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 247968us
I20260504 20:00:52.804602 14078 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:52.810254 14100 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:52.811105 14100 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:00:52.815515 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 69505.1us
I20260504 20:00:52.819041 12835 tablet_copy_service.cc:342] P 5ff0ef72ec3b4667893a0420d3c7f923: Request end of tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-0e8eba4dbb8d436f81062ca3bbf46788 received from {username='slave'} at 127.0.0.1:48774
I20260504 20:00:52.819788 12835 tablet_copy_service.cc:434] P 5ff0ef72ec3b4667893a0420d3c7f923: ending tablet copy session ef6057d9a49e4942b47ea0fcd66ab6f7-0e8eba4dbb8d436f81062ca3bbf46788 on tablet 0e8eba4dbb8d436f81062ca3bbf46788 with peer ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:52.820622 14100 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:52.823592 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:52.825930 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:52.826516 12816 raft_consensus.cc:3055] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:52.826800 12816 raft_consensus.cc:740] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:52.827467 12816 consensus_queue.cc:260] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:52.828591 12816 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:52.892297 12816 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
W20260504 20:00:52.894315 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 62153.5us
I20260504 20:00:52.801928 14093 raft_consensus.cc:993] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:52.908039 12893 raft_consensus.cc:1275] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:52.939453 12890 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "36b69b3d13384774a38b1442833e6948"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:00:52.942044 14028 raft_consensus.cc:1081] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:52.943140 12895 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:52.947800 14132 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:52.949851 12890 raft_consensus.cc:606] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:00:52.969239 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 45254.2us
W20260504 20:00:52.997051 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 108365us
I20260504 20:00:52.962023 14136 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:53.010026 13529 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40940
I20260504 20:00:53.011003 13529 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:53.011515 13529 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:52.960868 14049 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:00:53.050942 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 53193.3us
I20260504 20:00:53.051837 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:53.052544 12749 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:53.050526 13925 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.055757 13925 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.033864 14058 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:00:53.058604 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 105931us
I20260504 20:00:53.054862 12646 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39332 (request call id 219) took 1835 ms. Trace:
W20260504 20:00:53.059855 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 138491us
I20260504 20:00:53.068315 13925 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.078574 13531 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:53.020267 14136 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:53.067433 14000 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.115762 13533 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:00:53.110939 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 49500.1us
I20260504 20:00:53.111892 12647 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36606 (request call id 115) took 1687 ms. Trace:
I20260504 20:00:53.113287 12749 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:53.067837 14058 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.084383 14099 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.103681 14088 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.133507 12821 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:53.115630 14055 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
W20260504 20:00:53.163717 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 85926.3us
I20260504 20:00:53.162957 14141 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.181766 14059 raft_consensus.cc:993] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: : Instructing follower 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:53.181766 14141 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:53.182541 14059 raft_consensus.cc:1081] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Signalling peer 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:53.121910 12647 rpcz_store.cc:276] 0504 20:00:51.425020 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:51.425211 (+ 191us) service_pool.cc:224] Handling call
0504 20:00:53.111825 (+1686614us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:00:51.729273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:00:51.736539 (+ 7266us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:00:51.736587 (+ 48us) write_op.cc:435] Acquired schema lock
0504 20:00:51.736613 (+ 26us) tablet.cc:662] Decoding operations
0504 20:00:51.738788 (+ 2175us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:00:51.738845 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0504 20:00:51.738898 (+ 53us) tablet.cc:685] Acquiring locks for 1 operations
0504 20:00:51.739141 (+ 243us) tablet.cc:701] Row locks acquired
0504 20:00:51.739172 (+ 31us) write_op.cc:260] PREPARE: finished
0504 20:00:51.739347 (+ 175us) write_op.cc:270] Start()
0504 20:00:51.739503 (+ 156us) write_op.cc:276] Timestamp: P: 1777924851739308 usec, L: 0
0504 20:00:51.739539 (+ 36us) op_driver.cc:348] REPLICATION: starting
0504 20:00:51.740388 (+ 849us) log.cc:844] Serialized 530 byte log entry
0504 20:00:52.960683 (+1220295us) op_driver.cc:464] REPLICATION: finished
0504 20:00:53.008621 (+ 47938us) write_op.cc:301] APPLY: starting
0504 20:00:53.008752 (+ 131us) tablet.cc:1367] starting BulkCheckPresence
0504 20:00:53.008886 (+ 134us) tablet.cc:1370] finished BulkCheckPresence
0504 20:00:53.009065 (+ 179us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:00:53.009388 (+ 323us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:00:53.009603 (+ 215us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 20:00:53.009678 (+ 75us) write_op.cc:312] APPLY: finished
0504 20:00:53.038803 (+ 29125us) log.cc:844] Serialized 23 byte log entry
0504 20:00:53.051063 (+ 12260us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:53.057252 (+ 6189us) write_op.cc:454] Released schema lock
0504 20:00:53.060029 (+ 2777us) write_op.cc:341] FINISH: Updating metrics
0504 20:00:53.085081 (+ 25052us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:53.100437 (+ 15356us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":47646,"apply.run_cpu_time_us":2370,"apply.run_wall_time_us":57012,"num_ops":1,"prepare.queue_time_us":244010,"prepare.run_cpu_time_us":3528,"prepare.run_wall_time_us":24559,"replication_time_us":1210008,"thread_start_us":99916,"threads_started":1}]]}
I20260504 20:00:53.130868 14000 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.060509 12646 rpcz_store.cc:276] 0504 20:00:51.214054 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:51.214302 (+ 248us) service_pool.cc:224] Handling call
0504 20:00:53.049751 (+1835449us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:00:51.500772 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:00:51.501687 (+ 915us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:00:51.501738 (+ 51us) write_op.cc:435] Acquired schema lock
0504 20:00:51.501767 (+ 29us) tablet.cc:662] Decoding operations
0504 20:00:51.540374 (+ 38607us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:00:51.553239 (+ 12865us) write_op.cc:641] Partition lock acquired for write op
0504 20:00:51.559269 (+ 6030us) tablet.cc:685] Acquiring locks for 1 operations
0504 20:00:51.574593 (+ 15324us) tablet.cc:701] Row locks acquired
0504 20:00:51.574619 (+ 26us) write_op.cc:260] PREPARE: finished
0504 20:00:51.574778 (+ 159us) write_op.cc:270] Start()
0504 20:00:51.574905 (+ 127us) write_op.cc:276] Timestamp: P: 1777924851574736 usec, L: 0
0504 20:00:51.574927 (+ 22us) op_driver.cc:348] REPLICATION: starting
0504 20:00:51.575625 (+ 698us) log.cc:844] Serialized 535 byte log entry
0504 20:00:52.010717 (+435092us) spinlock_profiling.cc:243] Waited 7.48 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:00:52.594818 (+584101us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b540030f5a0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c92c723 00007f015c930155 00007f015c9300e9 00007f015c930079 00007f015c930041 00007f015c92f33d 0000000000379254 0000000000379099 00007f0159910ab5 00007f0159912e01
0504 20:00:52.595213 (+ 395us) op_driver.cc:464] REPLICATION: finished
0504 20:00:53.006335 (+411122us) write_op.cc:301] APPLY: starting
0504 20:00:53.006445 (+ 110us) tablet.cc:1367] starting BulkCheckPresence
0504 20:00:53.006548 (+ 103us) tablet.cc:1370] finished BulkCheckPresence
0504 20:00:53.006571 (+ 23us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:00:53.006853 (+ 282us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:00:53.007027 (+ 174us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 20:00:53.007074 (+ 47us) write_op.cc:312] APPLY: finished
0504 20:00:53.007271 (+ 197us) log.cc:844] Serialized 23 byte log entry
0504 20:00:53.007461 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:53.007578 (+ 117us) write_op.cc:454] Released schema lock
0504 20:00:53.007640 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0504 20:00:53.008123 (+ 483us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:53.008174 (+ 51us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":405169,"apply.run_cpu_time_us":1773,"apply.run_wall_time_us":1772,"num_ops":1,"prepare.queue_time_us":259658,"prepare.run_cpu_time_us":9939,"prepare.run_wall_time_us":177640,"raft.queue_time_us":1358703,"raft.run_cpu_time_us":49553,"raft.run_wall_time_us":717042,"replication_time_us":1020104,"spinlock_wait_cycles":47768192,"thread_start_us":781335,"threads_started":4,"wal-append.queue_time_us":279073}]]}
I20260504 20:00:53.188704 14133 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260504 20:00:53.202834 14141 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:53.211898 13499 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:53.155536 14099 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.217195 14099 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.206254 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:53.153685 14088 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.239959 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:53.240602 14088 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:53.237179 14077 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:53.267549 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.245683 14099 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.283198 14000 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.302696 14077 raft_consensus.cc:697] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
W20260504 20:00:53.303154 14157 log.cc:927] Time spent T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260504 20:00:53.315966 14058 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.243932 13529 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:53.318423 13529 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.305847 14136 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.166813 14063 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:00:53.307371 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 63276.3us
I20260504 20:00:53.314608 14077 consensus_queue.cc:237] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:53.318464 14086 raft_consensus.cc:1064] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: attempting to promote NON_VOTER ef6057d9a49e4942b47ea0fcd66ab6f7 to VOTER
I20260504 20:00:53.321067 14099 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:53.330812 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 174314us
I20260504 20:00:53.337625 14000 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.422327 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:53.340901 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.427102 12816 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:53.421030 14000 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.429421 12817 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "36b69b3d13384774a38b1442833e6948"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:53102
I20260504 20:00:53.427584 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:53.437225 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:53.440537 12817 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:53.399240 14119 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:00:53.455396 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 171339us
I20260504 20:00:53.456111 12817 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:53.441535 12894 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:53.434762 12890 raft_consensus.cc:3055] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:53.469388 12890 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:53.470502 14000 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.471460 12890 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:53.473304 14149 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:00:53.474086 12890 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:53.480207 13499 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:53.472550 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:53.484268 12817 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.485728 12817 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:53.491961 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 57838.2us
I20260504 20:00:53.481402 12818 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:53.461921 14119 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.498687 13531 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:53.495734 14133 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.483891 14051 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:53.522080 12863 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:53.506608 14136 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:53.582775 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.587993 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:53.588910 12819 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:53.595091 12895 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:00:53.565502 14059 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:00:53.574913 14159 log.cc:927] Time spent T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.447s user 0.012s sys 0.000s
I20260504 20:00:53.605113 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.606312 12645 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39338 (request call id 224) took 1883 ms. Trace:
I20260504 20:00:53.595969 12895 raft_consensus.cc:606] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:53.537262 14133 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:53.621452 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.622169 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.619442 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.626727 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:53.607066 14099 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.632119 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.608328 12645 rpcz_store.cc:276] 0504 20:00:51.722757 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:51.722968 (+ 211us) service_pool.cc:224] Handling call
0504 20:00:53.606216 (+1883248us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:00:51.882209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:00:51.987335 (+105126us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:00:51.990899 (+ 3564us) write_op.cc:435] Acquired schema lock
0504 20:00:51.993424 (+ 2525us) tablet.cc:662] Decoding operations
0504 20:00:52.030332 (+ 36908us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:00:52.034496 (+ 4164us) write_op.cc:641] Partition lock acquired for write op
0504 20:00:52.048868 (+ 14372us) tablet.cc:685] Acquiring locks for 1 operations
0504 20:00:52.049192 (+ 324us) tablet.cc:701] Row locks acquired
0504 20:00:52.049227 (+ 35us) write_op.cc:260] PREPARE: finished
0504 20:00:52.049469 (+ 242us) write_op.cc:270] Start()
0504 20:00:52.049686 (+ 217us) write_op.cc:276] Timestamp: P: 1777924852049384 usec, L: 0
0504 20:00:52.049722 (+ 36us) op_driver.cc:348] REPLICATION: starting
0504 20:00:52.050781 (+ 1059us) log.cc:844] Serialized 530 byte log entry
0504 20:00:53.103356 (+1052575us) op_driver.cc:464] REPLICATION: finished
0504 20:00:53.134168 (+ 30812us) write_op.cc:301] APPLY: starting
0504 20:00:53.138672 (+ 4504us) tablet.cc:1367] starting BulkCheckPresence
0504 20:00:53.157695 (+ 19023us) tablet.cc:1370] finished BulkCheckPresence
0504 20:00:53.186368 (+ 28673us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:00:53.224318 (+ 37950us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:00:53.259792 (+ 35474us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 20:00:53.264911 (+ 5119us) write_op.cc:312] APPLY: finished
0504 20:00:53.291742 (+ 26831us) log.cc:844] Serialized 23 byte log entry
0504 20:00:53.321661 (+ 29919us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:53.321786 (+ 125us) write_op.cc:454] Released schema lock
0504 20:00:53.322637 (+ 851us) write_op.cc:341] FINISH: Updating metrics
0504 20:00:53.323160 (+ 523us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:53.323205 (+ 45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":32242,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":7684,"apply.run_cpu_time_us":2946,"apply.run_wall_time_us":191767,"num_ops":1,"prepare.queue_time_us":163,"prepare.run_cpu_time_us":5064,"prepare.run_wall_time_us":181351,"replication_time_us":1036009}]]}
I20260504 20:00:53.634666 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.644256 14088 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:53.647042 14088 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.647411 14119 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.655177 14166 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:53.656395 12820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:53.654832 14088 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:53.657610 12819 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:53.656124 14166 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.656136 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:53.658581 12820 raft_consensus.cc:606] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:53.661928 12746 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:00:53.668853 14171 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260504 20:00:53.672520 12746 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:53.692080 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.707727 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:00:53.711999 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:53.719154 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:53.734002 12746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:00:53.734712 12746 raft_consensus.cc:606] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:53.737704 14133 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:53.742916 14173 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:53.743577 14173 raft_consensus.cc:697] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:53.744086 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:53.747066 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:53.722394 14141 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:53.749594 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:53.749869 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260504 20:00:53.754221 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:53.755234 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:53.757373 14166 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:53.761386 12647 catalog_manager.cc:5671] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:53.759611 14173 consensus_queue.cc:237] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:53.766698 14174 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.153s user 0.002s sys 0.000s
W20260504 20:00:53.772250 14160 log.cc:927] Time spent T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.446s user 0.006s sys 0.000s
W20260504 20:00:53.747120 14151 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.228s user 0.000s sys 0.001s
I20260504 20:00:53.764946 12647 catalog_manager.cc:5671] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:00:53.785141 12890 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
W20260504 20:00:53.831853 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 269128us
I20260504 20:00:53.842370 13925 raft_consensus.cc:993] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:53.842986 13925 raft_consensus.cc:1081] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:53.852010 13497 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:53.810048 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:53.857568 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 336119us
I20260504 20:00:53.857743 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:53.858475 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:53.891845 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:53.893148 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:53.856190 14074 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
W20260504 20:00:53.906855 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:53.857705 14077 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:53.812069 14086 consensus_queue.cc:237] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:53.908972 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:53.912498 12896 raft_consensus.cc:3055] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:53.912820 12896 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:53.908979 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:53.922894 13529 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40924
I20260504 20:00:53.923733 13529 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:53.924224 13529 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:53.911577 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:53.921319 12896 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:53.930346 12896 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:53.937826 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:53.909703 14074 raft_consensus.cc:697] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:53.914595 14078 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:00:53.949685 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 409172us
I20260504 20:00:54.002005 12818 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:53.930980 14077 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:54.033785 14074 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:54.035211 12820 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:53.958814 14078 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:54.060941 13533 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:00:54.033907 14191 log.cc:927] Time spent T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
W20260504 20:00:54.099912 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:54.116861 14077 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:54.119436 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:54.122013 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:54.125072 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 167601us
W20260504 20:00:54.127063 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 160868us
W20260504 20:00:54.198036 14186 log.cc:927] Time spent T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
W20260504 20:00:54.198716 14172 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.248s user 0.000s sys 0.002s
I20260504 20:00:54.141505 14119 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:00:54.218838 14181 log.cc:927] Time spent T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
I20260504 20:00:54.296537 12896 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:54.245137 14119 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:54.327873 14168 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:54.432858 14166 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:54.438582 14166 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:54.451366 14168 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:00:54.825698 13529 consensus_meta.cc:287] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.896s user 0.010s sys 0.005s
I20260504 20:00:54.851867 14078 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:54.852500 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:54.833323 14180 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:00:54.852860 13533 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.020s sys 0.008s
W20260504 20:00:54.863611 13533 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s user 0.020s sys 0.008s
W20260504 20:00:54.865139 14194 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.012s sys 0.007s
W20260504 20:00:54.865746 14194 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.539s user 0.012s sys 0.007s
I20260504 20:00:54.870927 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:54.861619 14078 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:54.847021 13529 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:54.877941 14203 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:54.903139 14178 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:54.904500 14146 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:54.885426 14119 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:54.875273 14180 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:54.906250 14183 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:54.923959 14175 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:54.895633 14078 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:54.932824 14168 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:54.912704 14178 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:54.912014 14203 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:54.934980 14182 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:54.928704 14146 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:54.931314 14166 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:54.961020 14176 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:54.933049 14183 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:54.985100 14119 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.026865 14028 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.030582 14067 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.014120 14048 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:54.964494 14175 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:54.987051 14182 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.028913 14208 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.046429 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:55.048103 12899 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:55.039710 13529 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.039074 14209 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.043308 14051 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.045082 14094 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.036895 14119 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.043567 14048 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.033689 14028 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.015071 14176 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.033756 14067 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:55.052665 14208 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.166013 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:55.154217 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:55.167061 12894 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:00:55.089344 12899 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:55.079034 14183 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.080353 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1300) took 1277 ms. Trace:
W20260504 20:00:55.175078 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 125960us
I20260504 20:00:55.175689 14183 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.176309 13499 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:55.079155 14175 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.176918 14183 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.171870 14224 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.063482 13529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:40924 (request call id 239) took 1139 ms. Trace:
I20260504 20:00:55.177584 14168 raft_consensus.cc:2804] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:55.079607 14219 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.178292 14168 raft_consensus.cc:697] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:55.178037 14224 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.178381 14219 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.078328 14078 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.179353 14168 consensus_queue.cc:237] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:00:55.062515 14205 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260504 20:00:55.059114 14166 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.175621 12898 rpcz_store.cc:276] 0504 20:00:53.803081 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:53.805074 (+ 1993us) service_pool.cc:224] Handling call
0504 20:00:53.805526 (+ 452us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:00:53.806046 (+ 520us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:00:53.806090 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:00:53.812909 (+ 6819us) log.cc:844] Serialized 26 byte log entry
0504 20:00:54.466606 (+653697us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b1800250a20. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d471 00007f015c969def
0504 20:00:54.924888 (+458282us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:00:54.924934 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:54.925027 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:00:55.079936 (+154909us) raft_consensus.cc:1685] finished
0504 20:00:55.079967 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:55.080252 (+ 285us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":46111,"spinlock_wait_cycles":3671936,"thread_start_us":458046,"threads_started":1,"wal-append.queue_time_us":1157610}
I20260504 20:00:55.057265 14180 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.140257 14203 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.061894 14220 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.059391 14209 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:00:55.083289 14051 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.165002 14211 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:55.159075 14207 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.153741 14178 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.159355 14212 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:00:55.368563 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 192341us
I20260504 20:00:55.374414 14182 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.376192 14182 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:55.376727 14182 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.062577 14086 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:55.095952 14218 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.223024 14175 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.078274 14146 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.339519 14078 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.063120 14094 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.370267 14219 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.265611 14183 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.432029 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:55.352905 14176 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.353402 14206 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:55.239346 14196 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.159363 14215 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.340837 14224 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.439548 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1432) took 1280 ms. Trace:
I20260504 20:00:55.062280 14210 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:55.378822 14178 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.450903 12896 rpcz_store.cc:276] 0504 20:00:54.167475 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.438384 (+1270909us) service_pool.cc:224] Handling call
0504 20:00:55.438853 (+ 469us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:55.439065 (+ 212us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:00:55.439108 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:00:55.439174 (+ 66us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:00:55.439220 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:55.439311 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:55.439467 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:55.224677 13529 rpcz_store.cc:276] 0504 20:00:53.918928 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:53.919910 (+ 982us) service_pool.cc:224] Handling call
0504 20:00:55.058499 (+1138589us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:55.341682 14166 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.448237 14235 raft_consensus.cc:993] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: : Instructing follower ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:55.474612 14233 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:55.476239 14229 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:55.417295 14216 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.474779 14074 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.464643 14224 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:55.436555 14078 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.385622 14220 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.425582 14211 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.425151 14218 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.441359 14175 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.452229 14146 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.388609 14086 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.547771 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:55.378695 14207 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.559623 14207 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.475364 14233 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.543609 14235 raft_consensus.cc:1081] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Signalling peer ef6057d9a49e4942b47ea0fcd66ab6f7 to start an election
I20260504 20:00:55.466742 14178 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.548839 14223 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.381521 14212 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.553262 14048 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.467180 14196 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.552846 14182 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.459992 14215 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.582970 14217 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.549039 14224 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.565461 14146 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.476938 14229 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.579874 14028 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:55.550662 14074 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.595181 14048 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:55.580170 14067 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:55.478305 14206 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.612334 14208 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.534377 14210 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.563346 14216 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.612821 14146 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.602133 14217 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.603437 14051 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.684425 14236 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:00:55.686964 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 300980us
W20260504 20:00:55.687770 12791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.006s sys 0.003s
W20260504 20:00:55.688100 12791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s user 0.007s sys 0.003s
W20260504 20:00:55.688367 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 635349us
W20260504 20:00:55.569470 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 199149us
I20260504 20:00:55.689860 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:55.686165 14236 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:00:55.698274 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 255949us
I20260504 20:00:55.713146 12647 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39316 (request call id 220) took 2428 ms. Trace:
I20260504 20:00:55.701056 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:55.726413 14196 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:55.640285 14182 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.718626 12647 rpcz_store.cc:276] 0504 20:00:53.263424 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:53.277817 (+ 14393us) service_pool.cc:224] Handling call
0504 20:00:55.692052 (+2414235us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:00:53.828870 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:00:53.829721 (+ 851us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:00:53.829774 (+ 53us) write_op.cc:435] Acquired schema lock
0504 20:00:53.829830 (+ 56us) tablet.cc:662] Decoding operations
0504 20:00:53.830395 (+ 565us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:00:53.830498 (+ 103us) write_op.cc:641] Partition lock acquired for write op
0504 20:00:53.830541 (+ 43us) tablet.cc:685] Acquiring locks for 2 operations
0504 20:00:53.831001 (+ 460us) tablet.cc:701] Row locks acquired
0504 20:00:53.831027 (+ 26us) write_op.cc:260] PREPARE: finished
0504 20:00:53.831216 (+ 189us) write_op.cc:270] Start()
0504 20:00:53.831369 (+ 153us) write_op.cc:276] Timestamp: P: 1777924853831158 usec, L: 0
0504 20:00:53.831399 (+ 30us) op_driver.cc:348] REPLICATION: starting
0504 20:00:53.886300 (+ 54901us) log.cc:844] Serialized 973 byte log entry
0504 20:00:54.326190 (+439890us) op_driver.cc:464] REPLICATION: finished
0504 20:00:54.865527 (+539337us) write_op.cc:301] APPLY: starting
0504 20:00:54.865639 (+ 112us) tablet.cc:1367] starting BulkCheckPresence
0504 20:00:54.865754 (+ 115us) tablet.cc:1370] finished BulkCheckPresence
0504 20:00:54.865906 (+ 152us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:00:54.866255 (+ 349us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:00:54.866384 (+ 129us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 20:00:54.866446 (+ 62us) write_op.cc:312] APPLY: finished
0504 20:00:54.866722 (+ 276us) log.cc:844] Serialized 29 byte log entry
0504 20:00:54.866917 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:54.867012 (+ 95us) write_op.cc:454] Released schema lock
0504 20:00:54.867054 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0504 20:00:54.867972 (+ 918us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:54.867993 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1837,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":539053,"apply.run_cpu_time_us":1771,"apply.run_wall_time_us":2441,"num_ops":2,"prepare.queue_time_us":34301,"prepare.run_cpu_time_us":3929,"prepare.run_wall_time_us":57787,"replication_time_us":404682,"thread_start_us":573543,"threads_started":2}]]}
I20260504 20:00:55.676265 14211 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.644570 14208 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.679868 14086 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.638665 14048 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.712246 14236 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.753894 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:55.749955 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:55.756647 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:55.730175 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:55.756745 12823 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:55.680645 14233 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.755769 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:55.758785 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:55.759670 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:55.720984 14232 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:55.603138 14223 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.775876 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:55.771291 12897 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:55.778559 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:55.779520 12819 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:55.736651 14207 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.788015 14206 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.789824 14212 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.792925 14217 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:55.754961 14220 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:00:55.796200 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 107266us
I20260504 20:00:55.800145 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:00:55.802148 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:55.710381 14210 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.802595 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:55.772791 14245 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.771140 14224 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.771559 14078 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:55.710391 14243 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.749017 14182 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.800849 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:55.759619 14208 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:55.898000 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 136399us
W20260504 20:00:55.774784 12716 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.642s (SENT) [suppressed 39 similar messages]
W20260504 20:00:55.853930 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 48637.3us
I20260504 20:00:56.130290 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:56.130754 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 325199us
I20260504 20:00:56.131616 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:55.951179 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 59989.9us
I20260504 20:00:56.132505 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:56.134361 12746 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:55.834558 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.137995 14249 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:56.138563 14249 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: NON_VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: true } }
I20260504 20:00:56.141008 12824 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:56.141214 14249 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:55.807237 14175 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.774011 14229 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:56.118800 14178 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.816051 14233 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.915906 14065 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:55.901610 14166 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:55.801108 14207 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.799271 14232 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:55.925992 13524 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:40916
I20260504 20:00:56.188598 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:56.487385 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 366364us
I20260504 20:00:55.844091 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.509065 13524 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:56.500957 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:56.510324 13524 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:56.510866 12897 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:56.511221 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.512275 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:56.513273 12897 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:00:56.451644 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.473421 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:56.515435 12824 raft_consensus.cc:3055] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:56.515545 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.515892 12824 raft_consensus.cc:740] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:56.516407 12863 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:56.516607 12742 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
W20260504 20:00:56.453963 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:00:56.318496 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 466011us
I20260504 20:00:56.245853 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:00:56.555337 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 46254.4us
I20260504 20:00:56.520601 13532 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:56.518764 12863 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:56.447800 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.498374 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:56.522862 14176 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:56.243043 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.446973 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:55.917093 14215 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.461185 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.569360 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:56.137007 14245 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.568008 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:56.520812 14086 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.008s sys 0.016s
I20260504 20:00:55.968664 14096 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:56.578298 14247 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:56.522835 14168 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:56.574335 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:55.957705 14236 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:56.446918 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:55.969111 14237 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:56.576922 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.374933 14187 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:55.950490 14074 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.521508 14078 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:56.521942 14067 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:56.583298 12750 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:56.450742 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.517556 12824 consensus_queue.cc:260] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.521035 14051 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.844429 14244 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:00:56.584163 14086 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.612s user 0.008s sys 0.021s
I20260504 20:00:55.916131 14146 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:56.595234 12897 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:56.595551 14236 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.596269 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.596726 14211 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:56.578644 14096 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.598137 14051 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.583567 14168 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:00:56.594296 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 75834.6us
I20260504 20:00:56.604327 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.348117 14048 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.591569 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.584501 14247 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:56.299182 14208 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.607900 12715 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:55.851069 14238 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:55.915076 14209 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:00:56.605592 12891 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:55.924366 14216 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.373979 14223 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.609566 12750 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:56.587134 14237 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:55.968915 14241 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:56.615612 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.146929 14243 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.602972 14176 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.618080 12750 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:56.619652 14051 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.619944 14176 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.616936 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.610215 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.341636 14218 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:56.614974 14065 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.622726 12743 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:56.613229 14247 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:56.628681 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.630115 14168 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.601609 13524 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.632198 12746 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:56.633092 12897 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:56.620544 14067 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.624060 14146 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.637584 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1329) took 1603 ms. Trace:
I20260504 20:00:56.612558 14078 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.638114 12748 rpcz_store.cc:276] 0504 20:00:55.034515 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.034665 (+ 150us) service_pool.cc:224] Handling call
0504 20:00:55.035003 (+ 338us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:55.056025 (+ 21022us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:00:55.056057 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:00:55.056099 (+ 42us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:00:55.056136 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:56.637073 (+1580937us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b5400242554. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:00:56.637161 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:56.637371 (+ 210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47741824}
I20260504 20:00:56.640734 14236 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:00:56.641453 12791 connection.cc:254] RPC call timeout handler was delayed by -1.5380039920000854s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20260504 20:00:56.620397 14211 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.639380 14182 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.641312 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1317) took 1212 ms. Trace:
I20260504 20:00:56.639694 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:56.622756 14220 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260504 20:00:56.646643 12791 connection.cc:254] RPC call timeout handler was delayed by -1.3589039920000232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1757s for a response
I20260504 20:00:56.644153 14176 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:56.647248 14067 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.641438 14078 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.656891 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53094 (request call id 506) took 3094 ms. Trace:
I20260504 20:00:56.658500 14233 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.647368 14245 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.650205 12898 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260504 20:00:56.644330 12789 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.788s user 0.010s sys 0.011s
I20260504 20:00:56.648886 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:56.651556 12791 connection.cc:254] RPC call timeout handler was delayed by -1.2833039920001283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
W20260504 20:00:56.664467 12791 connection.cc:254] RPC call timeout handler was delayed by -1.2266039920000367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
W20260504 20:00:56.664902 12791 connection.cc:254] RPC call timeout handler was delayed by -1.2032039920001125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20260504 20:00:56.665230 12791 connection.cc:254] RPC call timeout handler was delayed by -1.1738552510000773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
W20260504 20:00:56.665591 12791 connection.cc:254] RPC call timeout handler was delayed by -1.0728370780000205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
W20260504 20:00:56.665946 12791 connection.cc:254] RPC call timeout handler was delayed by -1.0169039920000387s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
W20260504 20:00:56.666273 12791 connection.cc:254] RPC call timeout handler was delayed by -1.0061039920001349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
I20260504 20:00:56.668606 12747 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:56.669098 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:56.652217 14176 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:56.654125 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.639324 14146 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.648073 12816 rpcz_store.cc:276] 0504 20:00:55.429203 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.429386 (+ 183us) service_pool.cc:224] Handling call
0504 20:00:55.429737 (+ 351us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:55.560433 (+130696us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:00:55.560481 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:00:55.560542 (+ 61us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:00:55.560588 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:56.640850 (+1080262us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54003007d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:00:56.640917 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:56.641111 (+ 194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":299960192}
I20260504 20:00:56.667138 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.657398 12822 rpcz_store.cc:276] 0504 20:00:53.561934 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:53.562058 (+ 124us) service_pool.cc:224] Handling call
0504 20:00:56.654878 (+3092820us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54000e3dd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c918470 00007f015c9b731a 00007f016198611e 00007f015af02437 00007f015af023c2 00007f015af022f1 00007f015af0226c 00007f015af01522 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:00:56.656610 (+ 1732us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2997205248}
I20260504 20:00:56.654023 14220 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:56.660184 14187 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.682832 14211 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:00:56.670096 12890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 35) took 1652 ms (client timeout 1663 ms). Trace:
W20260504 20:00:56.680065 12789 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.890s user 0.010s sys 0.011s
I20260504 20:00:56.683228 14176 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:56.698704 14178 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:56.699774 14187 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.699535 14178 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:56.696893 12890 rpcz_store.cc:269] 0504 20:00:55.017375 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:56.668450 (+1651075us) service_pool.cc:224] Handling call
0504 20:00:56.670004 (+ 1554us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:56.687180 14078 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:56.687753 12821 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
W20260504 20:00:56.688074 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 874091us
I20260504 20:00:56.691077 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.691548 12741 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:56.691972 13524 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.694041 13529 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 LEARNER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:56.695497 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.686053 14220 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:56.678323 14096 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.722990 14178 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.744678 12819 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:00:56.723268 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.746140 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.746338 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.747885 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.727339 14243 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.721751 12746 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:56.730482 14177 raft_consensus.cc:993] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: : Instructing follower 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:56.731065 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.734367 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1442) took 1042 ms. Trace:
I20260504 20:00:56.735215 13528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 241) took 1042 ms. Trace:
I20260504 20:00:56.769780 14210 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:56.770069 13528 rpcz_store.cc:276] 0504 20:00:55.692498 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.699666 (+ 7168us) service_pool.cc:224] Handling call
0504 20:00:56.690666 (+991000us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:00:56.735111 (+ 44445us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31398656}
I20260504 20:00:56.770503 14210 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.771613 13529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 240) took 1079 ms. Trace:
I20260504 20:00:56.772109 13529 rpcz_store.cc:276] 0504 20:00:55.691857 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.699619 (+ 7762us) service_pool.cc:224] Handling call
0504 20:00:56.692959 (+993340us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d3adc 00007f01598d3a46 000000000035a3c9 00007f016190b8e0 00007f01619d4784 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:00:56.693274 (+ 315us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:56.729305 (+ 36031us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:56.771519 (+ 42214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4462976}
I20260504 20:00:56.774652 14236 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:56.774739 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.767197 14177 raft_consensus.cc:1081] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Signalling peer 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:56.767971 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.768627 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.768975 14237 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.765491 12893 rpcz_store.cc:276] 0504 20:00:55.679341 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:56.674365 (+995024us) service_pool.cc:224] Handling call
0504 20:00:56.701583 (+ 27218us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:56.718332 (+ 16749us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:00:56.718369 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:00:56.718458 (+ 89us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:00:56.718523 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:56.718586 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:56.718742 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:00:56.743500 14146 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:56.705380 14257 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:56.631632 14241 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.775537 14236 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.812868 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.813454 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.814503 14257 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.801714 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 244) took 1052 ms. Trace:
I20260504 20:00:56.817162 14167 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:56.650485 14244 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.824319 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.829629 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.830325 12746 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:56.827368 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.831970 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1330) took 1797 ms. Trace:
I20260504 20:00:56.832211 14241 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.826576 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.832898 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.810185 14065 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.832629 14094 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.832449 12744 rpcz_store.cc:276] 0504 20:00:55.034887 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.035038 (+ 151us) service_pool.cc:224] Handling call
0504 20:00:55.035416 (+ 378us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:55.229050 (+193634us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:00:55.229086 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:00:55.229134 (+ 48us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:00:55.229168 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:56.820280 (+1591112us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b5400027454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:00:56.822162 (+ 1882us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:56.831748 (+ 9586us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":444829440}
I20260504 20:00:56.855387 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.819375 14238 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.821434 14146 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.862006 14232 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:00:56.865991 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 49921.2us
I20260504 20:00:56.870594 14257 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.870826 14236 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.871989 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.872927 13528 raft_consensus.cc:2393] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:00:56.873927 12816 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:56.875046 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.876384 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.818631 13527 rpcz_store.cc:276] 0504 20:00:55.749299 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.753915 (+ 4616us) service_pool.cc:224] Handling call
0504 20:00:56.705689 (+951774us) spinlock_profiling.cc:243] Waited 8.73 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3a3e 000000000035a3c9 00007f016190b8e0 00007f01619d4784 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:00:56.782460 (+ 76771us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20083968}
I20260504 20:00:56.881330 14146 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:56.883296 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1320) took 1452 ms. Trace:
I20260504 20:00:56.884164 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.884757 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.886521 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:56.887351 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.882340 12821 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:00:56.888878 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.890182 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.915867 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.877501 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.892876 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.856217 14233 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:56.919416 12818 rpcz_store.cc:276] 0504 20:00:55.430304 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.430480 (+ 176us) service_pool.cc:224] Handling call
0504 20:00:55.430823 (+ 343us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:55.613579 (+182756us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:00:55.613626 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:00:55.613683 (+ 57us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:00:55.613727 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:56.882576 (+1268849us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b540001d1d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:00:56.882666 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:56.882981 (+ 315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":419703168}
I20260504 20:00:56.927731 14207 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.884166 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.930169 12817 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:56.889634 14247 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.938647 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.924482 14210 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:56.929119 14238 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.946547 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.950510 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.948724 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:56.951762 14249 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:56.952363 13526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 339) took 1201 ms. Trace:
I20260504 20:00:56.953099 14247 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.952687 14249 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.974476 13499 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:56.976504 14249 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:56.976759 13499 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:56.960083 14220 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.943527 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:00:56.963629 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 36) took 1937 ms (client timeout 1506 ms). Trace:
I20260504 20:00:56.978730 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.964761 14074 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:56.873929 14167 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.967356 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:56.968403 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.971947 12863 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:56.945684 14195 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260504 20:00:56.973655 14238 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:56.987507 14238 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:56.959373 14260 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:56.978389 14112 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:00:56.974453 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:56.952980 13526 rpcz_store.cc:276] 0504 20:00:55.751274 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:55.758388 (+ 7114us) service_pool.cc:224] Handling call
0504 20:00:56.889167 (+1130779us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:00:56.952238 (+ 63071us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8643840}
W20260504 20:00:56.978853 12895 rpcz_store.cc:269] 0504 20:00:55.026228 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:56.963173 (+1936945us) service_pool.cc:211] Skipping call since client already timed out
0504 20:00:56.963374 (+ 201us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 20:00:56.985317 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.991796 12821 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:56.993602 14238 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:56.997910 12817 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:00:57.002733 12823 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260504 20:00:57.010816 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 43618.4us
I20260504 20:00:57.002045 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:57.012413 12791 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.014124 14028 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:57.014688 14028 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:57.015102 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1325) took 1583 ms. Trace:
I20260504 20:00:57.015539 12825 rpcz_store.cc:276] 0504 20:00:55.432001 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:56.652524 (+1220523us) service_pool.cc:224] Handling call
0504 20:00:56.653105 (+ 581us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:56.826040 (+172935us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:00:56.826073 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:00:56.826127 (+ 54us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:00:56.826153 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:57.014700 (+188547us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400376754. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:00:57.014744 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:57.014926 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397192576}
I20260504 20:00:57.015436 12646 catalog_manager.cc:5671] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:57.017335 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:56.985895 14244 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.028810 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.032670 14119 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.038153 14176 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.040351 12745 raft_consensus.cc:2393] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 2: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:00:57.040513 14176 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.041211 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.042467 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.042984 14220 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.008857 13533 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 330) took 3057 ms. Trace:
I20260504 20:00:57.015131 14028 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.020514 14218 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:57.023494 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:56.988562 14260 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:56.984696 14074 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:57.000017 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.005535 12819 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:00:57.047380 13525 raft_consensus.cc:1240] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.048018 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.044559 13533 rpcz_store.cc:276] 0504 20:00:53.951162 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:53.952220 (+ 1058us) service_pool.cc:224] Handling call
0504 20:00:53.952514 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:54.060758 (+108244us) raft_consensus.cc:1534] Early marking committed up to index 3
0504 20:00:54.244956 (+184198us) log.cc:844] Serialized 14 byte log entry
0504 20:00:54.864110 (+619154us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:00:54.864168 (+ 58us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:00:54.864201 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:56.696694 (+1832493us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b180028be20. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598debe6 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9
0504 20:00:57.008366 (+311672us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54004b3f54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:00:57.008428 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:00:57.008644 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":96,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":4155,"spinlock_wait_cycles":262842112,"thread_start_us":618738,"threads_started":1,"wal-append.queue_time_us":602848,"wal-append.run_cpu_time_us":1194,"wal-append.run_wall_time_us":1001214}
I20260504 20:00:57.057562 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.049487 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.053097 12825 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:57.059300 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:57.059581 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:57.058622 12817 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:57.061064 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.062034 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.062924 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.064199 12816 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.064651 12743 raft_consensus.cc:2393] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 2: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:00:57.064471 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 52213.7us
I20260504 20:00:57.063598 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:57.063742 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.065963 13528 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.066231 12750 raft_consensus.cc:3055] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:57.066391 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:57.066687 12750 raft_consensus.cc:740] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:57.066244 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:57.059310 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:57.065699 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.069011 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.067672 12750 consensus_queue.cc:260] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.069955 12744 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:00:57.070389 12895 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.070588 12750 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.052481 14218 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:57.071274 14218 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.069500 12825 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:57.072384 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.064961 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.059125 14234 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:57.074254 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.074488 14234 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:57.074879 14234 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.083506 12818 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
W20260504 20:00:57.081146 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 104470us
I20260504 20:00:57.085888 12791 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.087522 12863 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:57.089893 12863 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:57.085402 12646 catalog_manager.cc:5671] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:57.096382 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.084501 14176 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:00:57.097388 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 45885.5us
I20260504 20:00:57.096865 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:57.090524 14236 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.102522 12791 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.107684 12791 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.106297 14179 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.108750 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.112084 14222 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:57.112627 14222 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:57.113045 14222 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.112221 14099 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.104267 14236 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.116986 14236 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.108541 14179 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.114494 14260 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.117527 14179 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.127502 12819 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:53102
I20260504 20:00:57.128161 12819 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:57.128453 12819 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.132341 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:57.121567 14229 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:57.134368 14229 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:57.134829 14229 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.135406 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:57.129727 14167 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.148919 13499 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:57.152375 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:57.153257 13524 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:57.137734 14176 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.161199 12819 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.161491 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:57.164811 12895 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:57.168149 12819 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.169209 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:57.169912 13499 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:57.174283 12823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 75) took 1004 ms. Trace:
I20260504 20:00:57.175702 13524 raft_consensus.cc:1240] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.176379 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.178717 14265 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.179049 12824 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.179229 14265 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:57.179706 14265 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.184715 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.175235 12823 rpcz_store.cc:276] 0504 20:00:56.170109 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:56.240830 (+ 70721us) service_pool.cc:224] Handling call
0504 20:00:56.964646 (+723816us) spinlock_profiling.cc:243] Waited 8.4 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:00:57.002207 (+ 37561us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:00:57.011718 (+ 9511us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:00:57.172938 (+161220us) spinlock_profiling.cc:243] Waited 34.7 ms on lock 0x7b5400180e50. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:00:57.174228 (+ 1290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99231232}
I20260504 20:00:57.188006 12895 raft_consensus.cc:1240] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:57.188091 13499 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:57.189214 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:57.190719 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:57.194048 14260 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:57.198274 12824 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:57.199863 13499 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:57.200191 12816 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:57.202199 14244 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:57.202021 14185 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:57.202646 14244 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:57.203111 14244 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.204989 12825 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.207020 12821 raft_consensus.cc:1240] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.188297 14168 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260504 20:00:57.214551 13499 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:57.222632 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.257068 13499 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:57.262778 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.256362 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.267802 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.269083 12743 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:57.268301 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.216336 14168 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:00:57.254930 14264 log.cc:927] Time spent T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260504 20:00:57.216606 13528 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:00:57.274305 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 47320.3us
I20260504 20:00:57.276566 12749 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.277631 13497 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:57.255252 14074 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:57.281147 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:57.284053 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 129628us
I20260504 20:00:57.280278 12790 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.289068 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.292526 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.285235 14051 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:57.296970 12715 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:57.308916 12864 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.314221 14062 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:57.315182 14167 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:57.317219 14225 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:57.318226 14225 raft_consensus.cc:697] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:57.319413 14225 consensus_queue.cc:237] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.324198 14266 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:57.290828 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.276723 14078 consensus_queue.cc:1059] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
W20260504 20:00:57.330344 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 128733us
I20260504 20:00:57.331735 12863 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:57.326404 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.334769 12749 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:57.297335 14195 raft_consensus.cc:2804] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:57.305634 14168 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.310401 14211 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:00:57.344836 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 60018.5us
I20260504 20:00:57.337855 14176 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.340454 14167 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:57.349721 14062 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260504 20:00:57.362617 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.362612 14229 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.365063 14229 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.366688 12790 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:00:57.367506 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 94556us
I20260504 20:00:57.344489 14168 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.345942 14195 raft_consensus.cc:697] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:57.374747 12750 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:57.375240 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:57.376668 12748 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.376883 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:57.377200 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:57.350180 14247 consensus_queue.cc:1059] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260504 20:00:57.382138 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.351395 14179 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.370047 14167 raft_consensus.cc:697] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:57.384716 13497 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:57.384029 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.373075 14168 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.373620 14213 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:57.382124 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:57.396725 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:57.418138 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.419057 14236 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.439774 14236 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.435763 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.446589 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.433765 12714 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.448791 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:00:57.453372 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 46299.7us
I20260504 20:00:57.455619 12748 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
W20260504 20:00:57.451758 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 73959.1us
I20260504 20:00:57.456171 12714 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:00:57.459089 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:57.461323 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.450016 14195 consensus_queue.cc:237] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.456575 14176 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.443920 14167 consensus_queue.cc:237] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.463742 14176 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:57.466424 14221 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:57.466521 14146 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:57.470876 14221 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:57.471364 14221 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.476271 14078 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:00:57.477670 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:00:57.468418 14267 log.cc:927] Time spent T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20260504 20:00:57.473474 14176 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.490877 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.474054 12714 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.510915 12821 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:57.511656 12714 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.489485 14217 raft_consensus.cc:3055] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:57.530431 12714 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.544660 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.508859 14269 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.546316 14099 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:57.549986 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.540575 14179 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.547170 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.553449 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:57.537220 14211 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.558377 12821 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.542232 14119 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.536522 14217 raft_consensus.cc:740] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:57.565474 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.567399 14183 raft_consensus.cc:2764] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260504 20:00:57.560559 14211 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:57.574572 14211 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.557313 14099 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.552990 14269 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.563047 14119 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:00:57.608665 14187 raft_consensus.cc:3055] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:00:57.610910 14187 raft_consensus.cc:740] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:57.612114 14187 consensus_queue.cc:260] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.600735 14269 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.613016 14222 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.601657 14119 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.613559 14187 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.622689 14179 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.623281 14249 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:00:57.617523 14218 consensus_meta.cc:287] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.546s user 0.009s sys 0.005s
I20260504 20:00:57.626713 14094 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:00:57.609100 14270 log.cc:927] Time spent T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.094s user 0.004s sys 0.000s
I20260504 20:00:57.629861 12821 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
W20260504 20:00:57.630481 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 77499.7us
I20260504 20:00:57.632344 13533 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.644740 12897 raft_consensus.cc:1240] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:57.643487 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.643559 12863 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:00:57.648906 12823 raft_consensus.cc:1240] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:57.641902 14218 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.656569 12817 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:57.645365 14179 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:57.672909 14234 consensus_meta.cc:287] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.577s user 0.008s sys 0.007s
I20260504 20:00:57.648584 14249 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.683436 14094 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.692256 14094 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.694600 14221 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.696465 13499 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:57.639863 14217 consensus_queue.cc:260] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.701237 12821 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.688716 12741 raft_consensus.cc:1275] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260504 20:00:57.706333 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:57.708606 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.713627 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.721509 14179 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.706358 14266 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:57.705524 14094 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:57.722266 14277 consensus_queue.cc:1059] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260504 20:00:57.722724 14094 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.723441 14222 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.724790 14094 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.725131 14249 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.714728 14217 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.731648 14211 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.699923 14279 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:57.687772 14234 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.743379 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.747500 12823 raft_consensus.cc:1240] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260504 20:00:57.703413 14244 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.749437 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.697862 14182 consensus_queue.cc:1059] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260504 20:00:57.752524 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:00:57.749126 14028 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.698s user 0.010s sys 0.005s
W20260504 20:00:57.763407 14274 log.cc:927] Time spent T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
W20260504 20:00:57.765201 14265 consensus_meta.cc:287] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.585s user 0.013s sys 0.003s
I20260504 20:00:57.765377 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:57.745376 14236 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.769598 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:57.769666 12825 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:57.762456 14218 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.784747 14119 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.772411 14279 raft_consensus.cc:697] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:57.788529 14234 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.789767 14221 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.790421 14179 raft_consensus.cc:3055] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:57.791020 14179 raft_consensus.cc:740] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:57.779444 14236 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.793079 14236 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.792064 14179 consensus_queue.cc:260] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.794631 14244 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.794869 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:57.765676 14265 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.796820 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:57.785188 14168 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.810477 13524 raft_consensus.cc:1240] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.777179 14211 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.768707 14028 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.813407 14265 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.813777 12895 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260504 20:00:57.814817 14028 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.818315 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:57.786767 14279 consensus_queue.cc:237] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.789857 14051 raft_consensus.cc:3055] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:57.819293 14051 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:57.819595 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.820128 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:57.820274 14051 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.794904 14179 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.822535 14078 consensus_queue.cc:1059] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:00:57.823535 14078 raft_consensus.cc:3055] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:00:57.823908 14078 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:57.824841 14078 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.816097 14236 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.831707 14176 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.833885 14269 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.835220 13526 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.836434 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:57.837539 14221 consensus_queue.cc:1059] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:00:57.838714 14221 raft_consensus.cc:3055] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:57.839099 14221 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:57.840024 14221 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.841234 13528 raft_consensus.cc:1240] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.841322 14168 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.842186 14074 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:57.842105 14168 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:00:57.842952 14074 consensus_queue.cc:1059] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260504 20:00:57.844038 14074 raft_consensus.cc:3055] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:00:57.844408 14074 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:57.846349 14078 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.834335 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:57.847575 14211 consensus_queue.cc:1059] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:57.849457 14211 raft_consensus.cc:3055] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:57.849839 14211 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:57.850740 14211 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.851238 14259 consensus_queue.cc:1059] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260504 20:00:57.852046 14211 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.852468 14259 raft_consensus.cc:3055] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:00:57.852878 14259 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:57.831871 14236 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.863132 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:57.864388 14236 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.839447 14119 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.864156 14259 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.821554 14051 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.865381 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:57.865697 14259 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.864578 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:57.862339 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:57.866492 12744 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.845448 14074 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.870460 14074 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.869463 13524 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:57.869524 12747 raft_consensus.cc:2393] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:00:57.809036 14099 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.841617 14221 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.875770 14099 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.878511 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.880653 14265 consensus_queue.cc:1059] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:00:57.881587 14247 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.882233 14247 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.886838 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:57.891173 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.892791 12744 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:00:57.901932 14176 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.910625 12865 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:57.918658 14281 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:57.919226 14281 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:57.919637 14281 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.919670 14269 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:57.918525 14277 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:00:57.920629 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 98964.1us
I20260504 20:00:57.930514 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:57.933507 14094 consensus_queue.cc:1059] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:00:57.934075 14277 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.934825 14094 raft_consensus.cc:3055] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:57.935242 14094 raft_consensus.cc:740] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:57.936470 14277 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:57.936134 14094 consensus_queue.cc:260] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.937506 14094 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:57.939136 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:57.941478 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.942309 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:57.943750 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:57.875783 14249 consensus_queue.cc:1059] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260504 20:00:57.948496 14195 raft_consensus.cc:3055] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:00:57.948681 14249 raft_consensus.cc:3055] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:00:57.949021 14195 raft_consensus.cc:740] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:57.949174 14249 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:57.897783 14167 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:57.950119 12864 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:57.951582 14195 consensus_queue.cc:260] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:57.951844 14249 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.952764 14195 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.953061 14284 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:57.953215 14249 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.952093 12747 raft_consensus.cc:1240] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260504 20:00:57.955705 14247 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:57.912659 14234 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:57.957625 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:57.913026 14062 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:57.958359 14062 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:57.957657 14099 raft_consensus.cc:2764] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260504 20:00:57.958818 14062 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:57.961706 12823 raft_consensus.cc:1240] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:57.963580 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:57.964395 14167 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:57.891287 14119 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260504 20:00:57.959790 14234 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:57.977062 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:57.979761 12745 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:00:57.980683 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:57.980788 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:57.978586 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:57.985962 12865 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:57.980159 13499 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:58.009060 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:58.010047 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:58.011862 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:58.012647 12821 raft_consensus.cc:2393] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:58.006165 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:58.017160 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.017289 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:58.018497 14217 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.020135 14225 consensus_queue.cc:1059] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260504 20:00:58.024335 14220 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:58.014669 14028 consensus_queue.cc:1059] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260504 20:00:58.014361 14260 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.027722 14229 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:58.024343 12648 catalog_manager.cc:5671] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:57.978713 14244 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
W20260504 20:00:58.032936 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 52864.8us
I20260504 20:00:58.025660 14220 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:00:58.040731 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 54522.3us
I20260504 20:00:58.041646 12791 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.043679 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.046403 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.047258 14222 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:58.046991 12791 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.049935 14222 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:58.038400 14220 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:58.052541 12791 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.054361 14222 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:58.046008 14247 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.064134 12897 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:58.064199 14062 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:58.066607 14062 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:58.070782 14236 raft_consensus.cc:3055] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:58.071215 14236 raft_consensus.cc:740] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:58.070497 12749 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:58.058765 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:58.076517 12865 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:58.079574 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:58.080443 14281 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:58.080318 14236 consensus_queue.cc:260] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:58.081444 12747 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:58.083926 14281 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:58.085112 12646 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36606 (request call id 116) took 4640 ms. Trace:
I20260504 20:00:58.085525 12646 rpcz_store.cc:276] 0504 20:00:53.445229 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:53.445437 (+ 208us) service_pool.cc:224] Handling call
0504 20:00:58.085013 (+4639576us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:00:57.322611 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:00:57.323352 (+ 741us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:00:57.323414 (+ 62us) write_op.cc:435] Acquired schema lock
0504 20:00:57.323447 (+ 33us) tablet.cc:662] Decoding operations
0504 20:00:57.329975 (+ 6528us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:00:57.344336 (+ 14361us) write_op.cc:641] Partition lock acquired for write op
0504 20:00:57.351146 (+ 6810us) tablet.cc:685] Acquiring locks for 2 operations
0504 20:00:57.451946 (+100800us) tablet.cc:701] Row locks acquired
0504 20:00:57.454648 (+ 2702us) write_op.cc:260] PREPARE: finished
0504 20:00:57.464357 (+ 9709us) write_op.cc:270] Start()
0504 20:00:57.479771 (+ 15414us) write_op.cc:276] Timestamp: P: 1777924857459435 usec, L: 0
0504 20:00:57.479859 (+ 88us) op_driver.cc:348] REPLICATION: starting
0504 20:00:57.480645 (+ 786us) log.cc:844] Serialized 907 byte log entry
0504 20:00:57.886481 (+405836us) op_driver.cc:464] REPLICATION: finished
0504 20:00:57.930500 (+ 44019us) write_op.cc:301] APPLY: starting
0504 20:00:57.930610 (+ 110us) tablet.cc:1367] starting BulkCheckPresence
0504 20:00:57.930748 (+ 138us) tablet.cc:1370] finished BulkCheckPresence
0504 20:00:57.930879 (+ 131us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:00:57.931424 (+ 545us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:00:57.931703 (+ 279us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 20:00:57.931765 (+ 62us) write_op.cc:312] APPLY: finished
0504 20:00:57.932070 (+ 305us) log.cc:844] Serialized 29 byte log entry
0504 20:00:57.932284 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:57.932377 (+ 93us) write_op.cc:454] Released schema lock
0504 20:00:57.932434 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0504 20:00:57.932827 (+ 393us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:57.932858 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":154040,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":43733,"apply.run_cpu_time_us":2309,"apply.run_wall_time_us":2310,"num_ops":2,"prepare.queue_time_us":112243,"prepare.run_cpu_time_us":38967,"prepare.run_wall_time_us":274600,"raft.queue_time_us":129321,"raft.run_cpu_time_us":35839,"raft.run_wall_time_us":75317,"replication_time_us":404192,"thread_start_us":229597,"threads_started":4,"wal-append.queue_time_us":128568}]]}
I20260504 20:00:58.090085 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:58.081845 14236 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:58.089569 12648 catalog_manager.cc:5671] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:58.077332 14179 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.091871 14220 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:58.092768 14099 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:00:58.089974 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:58.110976 12865 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.072140 14292 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:58.116369 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:58.118172 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.117069 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:58.104986 12749 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:58.079670 14167 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:58.124284 12817 raft_consensus.cc:1240] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260504 20:00:58.126955 13499 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:58.132725 13497 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:58.135057 12822 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:58.109627 14284 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.110338 14234 raft_consensus.cc:697] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:58.137766 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:58.103163 14260 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260504 20:00:58.148077 12865 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:58.148834 14183 raft_consensus.cc:3055] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:58.149284 14183 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:58.134364 14266 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:58.150327 14183 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:58.151327 14266 raft_consensus.cc:697] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:58.151715 14183 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:58.124055 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:58.149516 12865 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:58.118942 14099 raft_consensus.cc:697] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:58.165944 14167 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.165984 12747 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:58.154080 14074 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.168296 14099 consensus_queue.cc:237] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:58.129619 14182 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:58.130100 14247 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.173566 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:58.175782 12745 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:58.178349 14284 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.178767 12865 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:58.206519 12865 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:00:58.208153 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:58.215461 13499 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:00:58.222615 14247 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.207131 14259 consensus_queue.cc:1059] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260504 20:00:58.152396 14266 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:58.260106 12748 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:58.229411 14249 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.238854 14074 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:58.274690 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.275352 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.281456 12893 consensus_queue.cc:237] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:58.281692 12749 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:58.249780 14247 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:00:58.293841 12647 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39338 (request call id 225) took 1635 ms. Trace:
W20260504 20:00:58.294278 12635 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 91ff767ed7c24b97940a459421881055 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 82787d8e64d24ca885ca517aa8ed1be2 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:58.294512 12899 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260504 20:00:58.311501 14220 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:58.316761 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:58.300429 12647 rpcz_store.cc:276] 0504 20:00:56.652382 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:56.652785 (+ 403us) service_pool.cc:224] Handling call
0504 20:00:58.286973 (+1634188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1390,"reported_tablets":10}
I20260504 20:00:58.333577 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:58.333725 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.334542 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.335028 12645 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39332 (request call id 220) took 1416 ms. Trace:
I20260504 20:00:58.335469 12645 rpcz_store.cc:276] 0504 20:00:56.882430 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:56.888833 (+ 6403us) service_pool.cc:224] Handling call
0504 20:00:58.298827 (+1409994us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":8}
I20260504 20:00:58.297588 14234 consensus_queue.cc:237] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:58.334578 12899 raft_consensus.cc:2393] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:00:58.344015 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 67914us
I20260504 20:00:58.299721 14074 raft_consensus.cc:697] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
W20260504 20:00:58.356482 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:58.271477 14269 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260504 20:00:58.364730 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.365563 12899 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:00:58.365880 14173 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:58.375731 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.408852 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.409613 12899 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
W20260504 20:00:58.385563 12716 leader_election.cc:336] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.642s (SENT)
I20260504 20:00:58.413991 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.341548 14247 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:58.428205 12822 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:58.420356 12897 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
W20260504 20:00:58.445096 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:58.419458 12716 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:58.463927 14078 consensus_queue.cc:1059] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260504 20:00:58.464658 14168 raft_consensus.cc:3055] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:00:58.465081 14168 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:58.472036 14074 consensus_queue.cc:237] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:58.492300 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.505797 12897 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:58.514608 12818 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:58.520447 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.521296 12895 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:58.526057 12825 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
W20260504 20:00:58.528323 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 146166us
I20260504 20:00:58.529763 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.530529 12897 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
W20260504 20:00:58.533545 12716 leader_election.cc:336] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.513s (SENT)
I20260504 20:00:58.534238 12716 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:58.508773 12747 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:58.512580 14051 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:58.550217 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.550954 12894 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:58.552028 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.556802 12791 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:00:58.523849 14249 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260504 20:00:58.501784 14168 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:58.541224 14236 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:00:58.564817 12716 leader_election.cc:336] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.546s (SENT)
I20260504 20:00:58.555902 14286 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.609851 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.589754 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.594558 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:00:58.631141 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:58.629787 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.564481 14302 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:00:58.633069 12716 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:58.638024 12896 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:58.642513 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.672823 12892 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:58.643152 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.647936 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:58.655169 12893 raft_consensus.cc:2410] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
W20260504 20:00:58.682106 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 117317us
I20260504 20:00:58.613955 14236 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:58.687940 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.680181 12897 raft_consensus.cc:2376] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2. Re-sending same reply.
I20260504 20:00:58.695933 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.674827 14051 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.691423 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.716363 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:58.675460 14247 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:58.701069 12894 raft_consensus.cc:2410] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260504 20:00:58.680982 14302 raft_consensus.cc:697] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:58.688833 12896 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:58.707468 12749 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:58.705590 12895 raft_consensus.cc:1275] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:58.682534 12890 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:58.721572 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.708029 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.723858 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:00:58.731871 14312 log.cc:927] Time spent T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
W20260504 20:00:58.737445 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:58.742235 12893 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:58.746968 12896 raft_consensus.cc:1240] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:58.749788 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:58.749766 12892 raft_consensus.cc:1240] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
W20260504 20:00:58.753989 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:58.736071 14269 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260504 20:00:58.750241 13526 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260504 20:00:58.773378 14236 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:58.775656 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:58.782816 14311 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.083s user 0.003s sys 0.000s
I20260504 20:00:58.788935 12820 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260504 20:00:58.790405 12891 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:00:58.790706 12898 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
W20260504 20:00:58.787565 14319 log.cc:927] Time spent T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20260504 20:00:58.795365 14094 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260504 20:00:58.809782 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:00:58.811005 14313 log.cc:927] Time spent T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.190s user 0.000s sys 0.002s
I20260504 20:00:58.822216 14302 consensus_queue.cc:237] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:58.797128 12791 leader_election.cc:336] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.558s (SENT)
I20260504 20:00:58.825660 12791 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:58.826745 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:58.827584 13498 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:58.827581 12891 raft_consensus.cc:2393] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:58.830215 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.830775 12891 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:00:58.818499 14281 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260504 20:00:58.835398 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.850248 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.850941 12898 raft_consensus.cc:2376] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2. Re-sending same reply.
I20260504 20:00:58.855746 12747 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:00:58.856726 12747 raft_consensus.cc:606] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:00:58.857481 12890 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:00:58.858219 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.848995 14309 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:00:58.565668 14074 consensus_queue.cc:1175] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260504 20:00:58.960487 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:58.961051 14168 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:00:58.869657 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 145569us
I20260504 20:00:58.868901 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.873441 13528 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:58.875277 12896 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:00:58.963471 12891 raft_consensus.cc:2376] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3. Re-sending same reply.
I20260504 20:00:58.963637 12896 raft_consensus.cc:606] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:58.964076 12896 raft_consensus.cc:614] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:00:58.878161 12647 catalog_manager.cc:5671] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 2 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:00:58.889991 14259 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:00:58.973207 12716 leader_election.cc:336] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.713s (SENT)
I20260504 20:00:58.973855 12716 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:58.920478 12645 catalog_manager.cc:5671] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:00:58.873582 14266 raft_consensus.cc:3055] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:00:58.930241 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:58.938977 12897 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:58.865396 13526 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:58.949244 12741 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:58.953786 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:58.992633 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:58.954193 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:58.955627 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:58.965322 12647 catalog_manager.cc:5671] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:00:58.991284 14294 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:58.977121 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:58.996832 12647 catalog_manager.cc:5671] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:58.999133 12825 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:58.999262 14195 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:00:58.999883 12825 raft_consensus.cc:606] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.001890 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:58.978755 14266 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:59.003474 14195 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:00:59.004220 12825 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:00:59.004839 12825 raft_consensus.cc:606] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Received request to transfer leadership to TS ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.005213 12825 raft_consensus.cc:614] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:00:58.986510 14259 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.008509 14259 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.011168 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.012123 12898 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:00:59.011649 14266 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.014079 12745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:00:59.014726 12745 raft_consensus.cc:606] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:00:58.994658 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 197552us
I20260504 20:00:58.975520 12645 catalog_manager.cc:5671] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:00:59.017871 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:59.017460 12645 catalog_manager.cc:5671] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:59.013574 14266 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:00:59.033458 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 60288.7us
I20260504 20:00:59.034448 12645 catalog_manager.cc:5671] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
W20260504 20:00:59.042032 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:59.044275 13499 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:59.047622 12742 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:59.040060 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260504 20:00:59.051599 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.051441 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.050305 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:00:59.057859 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.061767 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.062760 12899 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:00:59.064010 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.057099 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.068233 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.068765 12750 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:00:59.070060 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.065608 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.075670 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.083184 12742 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:00:59.073221 12899 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:00:59.085913 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.087136 12892 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:59.054214 14249 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:59.058504 14187 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:00:59.089421 12742 raft_consensus.cc:1240] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.093304 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.089010 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.094275 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.096166 12791 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.098897 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:00:59.099097 12791 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.100333 12791 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.101039 14099 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:59.101459 14099 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:59.088677 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.102053 14099 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.102396 14291 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.103179 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:59.104314 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.105437 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.057598 14207 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:59.114022 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.107272 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.107575 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.108132 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.115708 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.109082 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.109727 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.102213 14195 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:59.116564 12896 raft_consensus.cc:2393] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 2: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:59.113641 14207 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.106968 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.096827 14187 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:59.118472 14187 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:00:59.118899 14195 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.121052 14195 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:59.123025 14249 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.124282 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.124936 12749 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:00:59.126595 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:59.127348 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:59.089468 14244 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260504 20:00:59.127384 13528 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:00:59.130311 12790 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:59.131160 14244 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:59.131520 14244 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:59.131886 14244 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:59.132326 13499 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:59.133227 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.134035 12899 raft_consensus.cc:2393] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 2: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:59.134655 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.134825 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.123037 14207 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.135313 12741 raft_consensus.cc:1275] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:59.136053 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.136859 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.138101 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.138341 14094 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:59.139932 14320 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:00:59.140569 12745 raft_consensus.cc:1240] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.141376 12743 raft_consensus.cc:1240] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.152755 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.137096 12898 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:59.141520 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.153373 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:59.141850 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.101369 12791 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.144865 12892 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.147169 12744 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:00:59.146994 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.154974 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.101984 14218 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.137691 12891 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:00:59.125509 14249 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:59.143424 12825 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:59.157943 12743 raft_consensus.cc:1240] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.162272 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:59.162834 12892 raft_consensus.cc:1240] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.164183 12892 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.168922 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.170724 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.175357 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:59.184067 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.186409 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.188973 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.151284 14320 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:00:59.174041 14207 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:59.203251 14207 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:59.203814 14207 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.197578 14195 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:59.204649 14195 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.206719 14195 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:00:59.207269 14195 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:59.208587 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:00:59.209321 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 108014us
I20260504 20:00:59.210063 12791 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.207731 14195 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.194583 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.196120 14320 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:00:59.208742 14221 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260504 20:00:59.225121 12791 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.225287 14292 raft_consensus.cc:2804] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:59.226642 14244 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.236320 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.225734 14292 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:59.242702 14292 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.243492 14218 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.266965 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:59.268177 12716 leader_election.cc:336] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.647s (SENT)
I20260504 20:00:59.268694 12716 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:00:59.272444 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 47415.1us
I20260504 20:00:59.274601 12897 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260504 20:00:59.277447 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.283353 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.297667 14187 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.282056 14099 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.302556 13528 raft_consensus.cc:2955] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:59.306593 12824 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.305780 14099 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.311406 14294 consensus_queue.cc:1059] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:59.312793 14183 raft_consensus.cc:3055] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:59.312906 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:59.313252 14183 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:59.313572 12895 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:00:59.314128 14183 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.315404 14183 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.322335 14187 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.286834 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.326514 12895 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:00:59.328277 13499 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:59.281023 14051 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:00:59.336249 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 63106.8us
I20260504 20:00:59.334867 14187 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:00:59.337785 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.338095 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.340516 12895 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:00:59.341426 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:59.351612 12791 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.354554 14187 raft_consensus.cc:697] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:59.355602 14187 consensus_queue.cc:237] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.358884 14218 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.361019 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:59.345503 14328 log.cc:927] Time spent T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260504 20:00:59.367096 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.368129 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.369493 12893 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:59.371517 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:00:59.372951 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:00:59.375701 12716 leader_election.cc:336] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.992s (SENT)
I20260504 20:00:59.376150 12716 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.383387 12791 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.384029 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.386138 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.386504 12890 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:00:59.386883 14229 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:59.387567 12892 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:59.384639 12896 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:00:59.393790 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:00:59.394692 12895 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:00:59.395123 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.396436 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:00:59.397949 12865 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:00:59.398332 12865 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:00:59.398677 12865 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:59.400506 12716 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.402390 14065 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.402911 14099 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:59.403177 14065 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.403394 14099 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:59.403807 14099 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.405138 14294 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.383137 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.434830 14183 raft_consensus.cc:993] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: : Instructing follower 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:59.439888 12817 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:00:59.440407 14183 raft_consensus.cc:1081] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Signalling peer 5ff0ef72ec3b4667893a0420d3c7f923 to start an election
I20260504 20:00:59.442704 14065 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.443853 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:59.437340 14221 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:59.447968 14094 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:37306 (request call id 335) took 1169 ms. Trace:
I20260504 20:00:59.448302 14294 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.448475 14094 rpcz_store.cc:276] 0504 20:00:58.278664 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:58.278821 (+ 157us) service_pool.cc:224] Handling call
0504 20:00:58.329487 (+ 50666us) log.cc:844] Serialized 487 byte log entry
0504 20:00:58.649287 (+319800us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b5400214420. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c92ef06 00007f015c92887a 00007f015c927af4 00007f015c90828a 00007f015c90f133 00007f015c90f0b9 00007f015c90f049 00007f015c90f011 00007f015c90e1dd 0000000000379254
0504 20:00:58.778947 (+129660us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b5400214420. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c927522 00007f015c90828a 00007f015c90f133 00007f015c90f0b9 00007f015c90f049 00007f015c90f011 00007f015c90e1dd 0000000000379254 0000000000379099 00007f0159910ab5
0504 20:00:58.782130 (+ 3183us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:00:58.926167 (+144037us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b5400214420. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c92419e 00007f015c92ed10 00007f015c92ecb8 00007f015c92ec38 00007f015c92ebf3 00007f015c92de7c 00007f0162806d16 00007f01628014c8 00007f015c96abab 00007f015c96d6a0
0504 20:00:59.447625 (+521458us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6311,"raft.queue_time_us":6270,"raft.run_cpu_time_us":5382,"raft.run_wall_time_us":427269,"spinlock_wait_cycles":85218048,"thread_start_us":142045,"threads_started":1,"wal-append.queue_time_us":240323}
I20260504 20:00:59.451493 14247 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260504 20:00:59.451975 14329 log.cc:927] Time spent T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260504 20:00:59.453702 12746 raft_consensus.cc:1240] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.438637 12716 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.438524 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.454321 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 593a993b4ff341c9b98629e2c051ce00 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:00:59.436136 14244 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.458671 12865 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.466773 14218 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:00:59.467473 14218 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:59.468047 14218 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.467867 14207 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.470299 12820 raft_consensus.cc:1240] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.447556 14195 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:59.482208 14326 log.cc:927] Time spent T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.100s user 0.018s sys 0.001s
I20260504 20:00:59.482443 14292 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.483435 14294 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.484242 12821 raft_consensus.cc:1240] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
W20260504 20:00:59.489037 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 87608.7us
I20260504 20:00:59.489675 12791 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.493095 12821 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:53098
I20260504 20:00:59.493261 14225 consensus_queue.cc:1059] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:59.493884 12821 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:00:59.494359 12821 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.494990 14225 raft_consensus.cc:3055] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:59.495713 14225 raft_consensus.cc:740] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:59.492291 14146 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:59.498631 14146 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.500118 14182 consensus_queue.cc:1059] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:59.492012 14051 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:59.502530 14051 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.503484 14099 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.504639 14229 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:00:59.505637 14229 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:00:59.509027 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 70476.4us
W20260504 20:00:59.509745 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 51186.1us
I20260504 20:00:59.509893 12716 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.510602 12865 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.510684 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:59.511729 12825 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:59.502170 14182 raft_consensus.cc:3055] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:59.516217 14182 raft_consensus.cc:740] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:59.516711 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:59.517419 13533 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:00:59.517787 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:59.517621 14182 consensus_queue.cc:260] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.516376 14247 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.518756 12715 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:59.514919 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:59.519909 14183 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:59.520363 14183 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:59.520205 13524 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:00:59.520918 14183 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:59.522027 12715 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:59.524854 14065 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:00:59.525254 14065 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:59.525635 14065 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:59.505718 14051 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.527504 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:59.528666 12817 raft_consensus.cc:2393] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:59.519587 14146 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.497179 14225 consensus_queue.cc:260] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.532189 14225 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.534085 14229 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.534610 14320 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.538362 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.539068 12747 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:00:59.540053 14182 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.540532 14207 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.543087 14320 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:00:59.544116 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:59.485683 14292 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.556428 14309 consensus_queue.cc:1059] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:59.558218 14309 consensus_queue.cc:1059] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:59.560734 14302 raft_consensus.cc:3055] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:59.561180 14247 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.561529 14302 raft_consensus.cc:740] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:59.562987 14302 consensus_queue.cc:260] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:00:59.563647 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 53126.8us
I20260504 20:00:59.564604 14302 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.564900 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.566202 14229 raft_consensus.cc:3055] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
W20260504 20:00:59.572649 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 62872.9us
I20260504 20:00:59.573570 12716 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.569336 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.575295 14182 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.576948 14247 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.578260 12865 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.577037 14335 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.581406 14335 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.584959 14293 raft_consensus.cc:993] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:00:59.585644 14292 raft_consensus.cc:1081] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:00:59.592932 12716 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.580607 12865 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.582424 12821 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.595476 14244 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.595831 14099 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.596652 12821 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.596891 12791 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.599951 14286 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.600966 14335 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.567870 14229 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:59.602947 14229 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.604066 14293 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:59.604765 14229 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:00:59.574641 14221 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57450 (request call id 564) took 1270 ms. Trace:
I20260504 20:00:59.606190 14221 rpcz_store.cc:276] 0504 20:00:58.279130 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:58.279648 (+ 518us) service_pool.cc:224] Handling call
0504 20:00:58.299325 (+ 19677us) log.cc:844] Serialized 487 byte log entry
0504 20:00:58.870762 (+571437us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b54002fd620. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c92ef06 00007f015c92887a 00007f015c927af4 00007f015c90828a 00007f015c90f133 00007f015c90f0b9 00007f015c90f049 00007f015c90f011 00007f015c90e1dd 0000000000379254
0504 20:00:58.927358 (+ 56596us) spinlock_profiling.cc:243] Waited 6.68 ms on lock 0x7b18002ea260. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598debe6 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159910a33 00007f0159912e01
0504 20:00:58.929495 (+ 2137us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b18002ea260. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159910a33 00007f0159912e01 00007f0159912db9 00007f0159912d49 00007f0159912d11 00007f015991200d 0000000000379254 0000000000379099 00007f01598f3fba
0504 20:00:59.549253 (+619758us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":246,"mutex_wait_us":47731,"raft.queue_time_us":33092,"raft.run_cpu_time_us":4052,"raft.run_wall_time_us":392561,"spinlock_wait_cycles":131826688,"thread_start_us":126069,"threads_started":1,"wal-append.queue_time_us":212395}
I20260504 20:00:59.612387 12747 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:00:59.603582 14218 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.614131 14183 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.624716 14218 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.625643 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:59.626863 12821 raft_consensus.cc:2393] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:00:59.630195 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 104156us
I20260504 20:00:59.615960 14065 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.617765 14293 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.594837 14225 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.619113 12716 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.620867 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.622334 12791 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.639322 12791 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.640287 13499 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:59.625559 14244 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:00:59.646543 14309 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.647210 14286 consensus_queue.cc:1059] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:59.648571 14218 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260504 20:00:59.649130 14286 raft_consensus.cc:3055] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:59.649739 14286 raft_consensus.cc:740] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:59.645296 14094 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.652791 12716 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.655040 12865 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.657537 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.642418 14221 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.670131 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:59.671075 12818 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:00:59.644441 14187 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:00:59.663719 14225 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.673419 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:59.675948 14187 raft_consensus.cc:697] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:00:59.665841 14065 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.666545 14183 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.667608 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:00:59.651010 14286 consensus_queue.cc:260] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.679785 14286 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.643573 14099 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.654052 14244 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.682118 12716 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.687809 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:59.677269 14187 consensus_queue.cc:237] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.640096 13526 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:00:59.700066 14244 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.701529 12790 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:00:59.704094 14218 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:00:59.704726 14218 raft_consensus.cc:697] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:00:59.675032 12865 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:00:59.707799 12865 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.708057 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:59.709024 12865 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.687970 12716 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.688551 12825 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:00:59.690132 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:59.710178 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.710346 12823 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:00:59.692606 12817 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.681660 14094 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.694216 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:59.715339 12820 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:00:59.715407 14094 consensus_queue.cc:1059] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:00:59.695179 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:00:59.696655 14207 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:59.716675 14207 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.717962 14236 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.718933 14207 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.698076 14225 raft_consensus.cc:993] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:00:59.686584 14302 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.720085 12824 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:00:59.723807 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:59.725266 14335 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.726459 14221 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.690295 14183 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.713514 14249 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.713737 14247 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.714186 14281 consensus_queue.cc:1059] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:59.745257 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.719094 14236 raft_consensus.cc:3055] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:59.708737 12818 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:00:59.729732 12716 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.737607 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.747779 12750 raft_consensus.cc:3055] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:59.748142 12750 raft_consensus.cc:740] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:59.737438 12817 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:00:59.743160 12823 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:00:59.743923 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:00:59.742103 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:00:59.752682 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.753543 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.752391 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.752357 14182 raft_consensus.cc:1081] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:00:59.752592 12750 consensus_queue.cc:260] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.775844 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.776723 12743 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
W20260504 20:00:59.777386 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:00:59.751231 14236 raft_consensus.cc:740] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:59.705974 14218 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.768556 12820 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:00:59.759106 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:00:59.787529 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:00:59.792824 12716 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:00:59.797586 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 85962.8us
I20260504 20:00:59.798419 12865 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.799628 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.799749 12817 raft_consensus.cc:1240] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:00:59.800812 12749 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:59.853823 12750 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.795614 14225 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
W20260504 20:00:59.855566 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 55487.3us
I20260504 20:00:59.855777 14259 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:00:59.855609 14221 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:00:59.847184 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 68408.1us
I20260504 20:00:59.858731 13525 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:00:59.859280 14249 raft_consensus.cc:3055] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
W20260504 20:00:59.860826 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 61270.1us
I20260504 20:00:59.919657 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.778854 14247 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:00:59.862547 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 69838.5us
I20260504 20:00:59.923135 12716 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:00:59.864418 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 61936.2us
I20260504 20:00:59.913727 12647 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39316 (request call id 222) took 1116 ms. Trace:
I20260504 20:00:59.794474 14183 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.873382 14281 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.922166 12865 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.919092 14249 raft_consensus.cc:740] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:59.925348 12750 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:59.928406 12716 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.948346 14345 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.928627 12715 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:59.949282 14236 consensus_queue.cc:260] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.950393 14146 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.951149 14146 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:00:59.951674 14146 raft_consensus.cc:697] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:00:59.952191 14236 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.952378 12895 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:00:59.952659 14146 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.929224 12748 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:00:59.684173 13528 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:00:59.860497 13524 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
W20260504 20:00:59.959709 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 60795.4us
I20260504 20:00:59.949329 12716 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:00:59.963581 12715 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:00:59.968984 12865 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.973479 14218 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:00:59.972544 12865 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:00:59.918502 14259 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.976823 14259 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:00:59.930578 14247 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:00:59.969971 14320 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260504 20:00:59.931375 14342 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:00:59.978384 14236 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.978888 14342 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.980793 14247 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:00:59.929705 14182 consensus_queue.cc:1059] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260504 20:00:59.987015 14249 consensus_queue.cc:260] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.988863 14249 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.992952 14345 consensus_queue.cc:1059] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:00:59.994285 14345 raft_consensus.cc:3055] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:00:59.924226 12647 rpcz_store.cc:276] 0504 20:00:58.797084 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:58.797291 (+ 207us) service_pool.cc:224] Handling call
0504 20:00:59.913638 (+1116347us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:00:59.109646 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:00:59.110461 (+ 815us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:00:59.110498 (+ 37us) write_op.cc:435] Acquired schema lock
0504 20:00:59.110518 (+ 20us) tablet.cc:662] Decoding operations
0504 20:00:59.113684 (+ 3166us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:00:59.113756 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0504 20:00:59.113788 (+ 32us) tablet.cc:685] Acquiring locks for 3 operations
0504 20:00:59.114240 (+ 452us) tablet.cc:701] Row locks acquired
0504 20:00:59.114261 (+ 21us) write_op.cc:260] PREPARE: finished
0504 20:00:59.114428 (+ 167us) write_op.cc:270] Start()
0504 20:00:59.114554 (+ 126us) write_op.cc:276] Timestamp: P: 1777924859114374 usec, L: 0
0504 20:00:59.114585 (+ 31us) op_driver.cc:348] REPLICATION: starting
0504 20:00:59.194792 (+ 80207us) log.cc:844] Serialized 1350 byte log entry
0504 20:00:59.330650 (+135858us) op_driver.cc:464] REPLICATION: finished
0504 20:00:59.448083 (+117433us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f01598f382c 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f01603f0360
0504 20:00:59.637921 (+189838us) write_op.cc:301] APPLY: starting
0504 20:00:59.638035 (+ 114us) tablet.cc:1367] starting BulkCheckPresence
0504 20:00:59.638299 (+ 264us) tablet.cc:1370] finished BulkCheckPresence
0504 20:00:59.638320 (+ 21us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:00:59.689931 (+ 51611us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:00:59.706086 (+ 16155us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0504 20:00:59.706190 (+ 104us) write_op.cc:312] APPLY: finished
0504 20:00:59.750286 (+ 44096us) log.cc:844] Serialized 35 byte log entry
0504 20:00:59.769249 (+ 18963us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:59.779438 (+ 10189us) write_op.cc:454] Released schema lock
0504 20:00:59.782267 (+ 2829us) write_op.cc:341] FINISH: Updating metrics
0504 20:00:59.782712 (+ 445us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:00:59.782749 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":118,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":302178,"apply.run_cpu_time_us":3475,"apply.run_wall_time_us":144774,"num_ops":3,"prepare.queue_time_us":108491,"prepare.run_cpu_time_us":4463,"prepare.run_wall_time_us":101187,"replication_time_us":170086,"spinlock_wait_cycles":6055680,"thread_start_us":318423,"threads_started":2}]]}
I20260504 20:00:59.994720 14345 raft_consensus.cc:740] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:00:59.995656 14345 consensus_queue.cc:260] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:00:59.996893 14345 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:00:59.998637 12817 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:00.000732 14183 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.931823 14294 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:00:59.932077 14094 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:00.010622 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 53403.2us
I20260504 20:01:00.014783 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:00.015450 12750 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:00:59.929571 14244 consensus_queue.cc:1048] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20260504 20:01:00.011904 12820 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:00.024201 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:00.027985 12895 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:01:00.019515 14281 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:00.029230 14281 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:00.020001 14183 consensus_queue.cc:1059] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260504 20:01:00.030789 14183 raft_consensus.cc:3055] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:00.031133 14183 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:00.031661 12741 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:00.016492 14065 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:00.032051 14281 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:00.032089 14183 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.044816 14339 raft_consensus.cc:3055] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:00.050976 12645 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39338 (request call id 226) took 1674 ms. Trace:
I20260504 20:01:00.020620 14286 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:00.061180 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:00.064015 14183 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:00.066167 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:00.057739 14339 raft_consensus.cc:740] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:00.069402 14339 consensus_queue.cc:260] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.071452 14339 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:00.072346 14286 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.077301 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:00.078406 12820 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:00.084033 14286 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:00.061291 12648 catalog_manager.cc:5671] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 2 to 3, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 3 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:01:00.091650 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 119215us
I20260504 20:01:00.093124 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.095232 12865 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.096966 14347 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:00.062391 14065 raft_consensus.cc:697] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:01:00.105230 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 130485us
W20260504 20:01:00.106598 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 143155us
I20260504 20:01:00.112287 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:00.112757 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:00.113409 13533 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:00.114584 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:00.115988 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:00.117228 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:00.117321 13499 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:00.118369 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:00.119184 12899 raft_consensus.cc:2393] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:00.122005 14348 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:00.122100 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:00.122777 14348 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:00.123346 14348 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:00.123474 13528 raft_consensus.cc:2410] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260504 20:01:00.124050 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:00.124871 12896 raft_consensus.cc:2393] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:00.126953 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:00.128562 12896 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:00.128789 12892 raft_consensus.cc:1275] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:00.102916 14344 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:00.106885 12648 catalog_manager.cc:5671] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: config changed from index 3 to 4, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:00.130822 12715 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:00.058342 12645 rpcz_store.cc:276] 0504 20:00:58.376836 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:58.399947 (+ 23111us) service_pool.cc:224] Handling call
0504 20:01:00.050877 (+1650930us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:00:59.233222 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:00:59.316144 (+ 82922us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:00:59.316194 (+ 50us) write_op.cc:435] Acquired schema lock
0504 20:00:59.316221 (+ 27us) tablet.cc:662] Decoding operations
0504 20:00:59.317028 (+ 807us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:00:59.317091 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0504 20:00:59.317133 (+ 42us) tablet.cc:685] Acquiring locks for 4 operations
0504 20:00:59.317640 (+ 507us) tablet.cc:701] Row locks acquired
0504 20:00:59.317661 (+ 21us) write_op.cc:260] PREPARE: finished
0504 20:00:59.317881 (+ 220us) write_op.cc:270] Start()
0504 20:00:59.318039 (+ 158us) write_op.cc:276] Timestamp: P: 1777924859317795 usec, L: 0
0504 20:00:59.318063 (+ 24us) op_driver.cc:348] REPLICATION: starting
0504 20:00:59.599248 (+281185us) log.cc:844] Serialized 1672 byte log entry
0504 20:00:59.602403 (+ 3155us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b540030f054. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9aa208 00007f01603ef9f1 00007f01603ef1c8 00007f01603f3e61 00007f01603f3e19 00007f01603f3da9 00007f01603f3d71 00007f01603f2f9d 0000000000379254 0000000000379099
0504 20:00:59.641711 (+ 39308us) op_driver.cc:464] REPLICATION: finished
0504 20:00:59.814246 (+172535us) write_op.cc:301] APPLY: starting
0504 20:00:59.820219 (+ 5973us) tablet.cc:1367] starting BulkCheckPresence
0504 20:00:59.859949 (+ 39730us) tablet.cc:1370] finished BulkCheckPresence
0504 20:00:59.859974 (+ 25us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:00.010581 (+150607us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:00.034367 (+ 23786us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0504 20:01:00.034443 (+ 76us) write_op.cc:312] APPLY: finished
0504 20:01:00.037048 (+ 2605us) log.cc:844] Serialized 41 byte log entry
0504 20:01:00.037297 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:00.037433 (+ 136us) write_op.cc:454] Released schema lock
0504 20:01:00.037495 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:00.037949 (+ 454us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:00.037991 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":137115,"apply.run_cpu_time_us":4196,"apply.run_wall_time_us":242489,"num_ops":4,"prepare.queue_time_us":117959,"prepare.run_cpu_time_us":9452,"prepare.run_wall_time_us":374637,"replication_time_us":318304,"spinlock_wait_cycles":644772096,"thread_start_us":75904,"threads_started":1}]]}
I20260504 20:01:00.138139 12750 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:00.138180 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:00.141512 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:00.142482 12864 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:00.142374 12648 catalog_manager.cc:5671] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:00.143676 12865 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.144416 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:00.143518 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:00.145694 12648 catalog_manager.cc:5671] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:00.147089 14094 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:00.146847 14281 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:00.150070 14229 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:00.146598 12894 raft_consensus.cc:1275] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:00.151770 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:00.152813 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:00.152968 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:00.151626 14229 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.164716 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:00.185186 12791 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:00.193920 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:00.144690 14286 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:00.201777 12895 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:00.153721 12890 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37318
I20260504 20:01:00.159338 14335 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:00.206300 12791 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:00.208541 12741 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:00.209465 12890 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:01:00.210161 12890 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:00.210958 14351 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:00.197293 14207 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:00.215679 14207 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:00.216349 14207 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:00.230204 14349 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:00.230576 12791 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:00.231236 14349 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:00.231611 12791 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:00.231875 14349 raft_consensus.cc:697] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:00.232641 12791 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:00.234455 14293 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:00.236754 13497 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.240229 12750 raft_consensus.cc:1275] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:00.241654 13497 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.204212 14286 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:00.263398 14281 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:00.126969 14065 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.282196 12791 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.243079 12898 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:00.260246 14266 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:00.257844 14099 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:00.285661 12791 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.275143 14320 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:00.266026 14218 raft_consensus.cc:2804] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:00.233130 14349 consensus_queue.cc:237] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.277736 14286 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:00.305756 12890 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.292708 12741 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:00.289544 14266 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:00.293023 14244 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:00.310472 14320 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:00.281337 14281 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.330235 12896 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:00.330163 12891 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:00.312467 14187 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260504 20:01:00.329771 14266 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:00.335932 14320 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:00.365221 12897 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:00.343566 14292 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:00.369992 14099 raft_consensus.cc:697] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:00.393437 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 125059us
W20260504 20:01:00.402895 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:00.358505 14302 consensus_queue.cc:1048] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260504 20:01:00.411777 14281 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:00.361239 14293 raft_consensus.cc:697] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:00.424062 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 129727us
I20260504 20:01:00.428330 14291 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:00.429112 14229 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:00.431505 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 482222us
I20260504 20:01:00.415956 14286 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.442982 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:00.433538 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:00.443692 12823 raft_consensus.cc:3055] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:00.444047 12823 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:00.437008 12890 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:00.437288 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:00.432991 12791 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.446785 12716 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:00.444762 12823 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.471035 14182 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:00.471446 12823 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:00.471608 12791 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.471726 14182 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.472754 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:00.469571 14353 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260504 20:01:00.473991 14364 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:00.474905 14366 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:00.475787 12716 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:00.477111 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:00.482405 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet dfa79aedaea6499facdfc14a3e7467f8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260504 20:01:00.483228 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:00.483911 12894 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:00.479344 14218 raft_consensus.cc:697] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:00.488431 14218 consensus_queue.cc:237] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.489085 12823 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:00.490761 12863 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:00.424722 14293 consensus_queue.cc:237] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.483232 14352 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:00.493065 14225 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:00.492879 14146 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:00.493533 14291 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:00.486537 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:00.495108 12890 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37334
I20260504 20:01:00.493463 14294 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:00.540449 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 179932us
I20260504 20:01:00.505945 14182 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:00.501004 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:00.510821 14351 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:00.485483 14355 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260504 20:01:00.528064 14352 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:00.588320 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 79416.1us
I20260504 20:01:00.557623 12890 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20260504 20:01:00.620638 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:00.624714 12791 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:00.559825 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:00.596310 14352 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:00.637426 12898 raft_consensus.cc:2955] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:00.554432 14294 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.627274 12820 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:00.647483 12823 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:00.625051 12890 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:00.558533 14361 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.087s
I20260504 20:01:00.659122 12817 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:00.660161 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:00.660822 12823 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:01:00.666841 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:00.552881 14099 consensus_queue.cc:237] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:00.642467 14360 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:00.646548 13528 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:00.681490 14360 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:00.690049 14362 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.306s user 0.001s sys 0.000s
I20260504 20:01:00.679134 14339 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:00.720098 14183 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.656s user 0.015s sys 0.000s
I20260504 20:01:00.690379 14351 raft_consensus.cc:697] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:00.685078 14355 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20260504 20:01:00.758574 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 63511.3us
I20260504 20:01:00.722532 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:00.731864 14360 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:00.774524 14371 log.cc:927] Time spent T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
W20260504 20:01:00.775681 14207 consensus_meta.cc:287] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.559s user 0.002s sys 0.014s
I20260504 20:01:00.786427 14351 consensus_queue.cc:237] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:00.790115 14363 log.cc:927] Time spent T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.250s user 0.001s sys 0.000s
I20260504 20:01:00.790647 12891 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:00.755586 14259 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:00.794095 14259 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.776252 14207 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:00.816319 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:00.808840 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:00.819196 12822 raft_consensus.cc:2393] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:00.823447 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:00.824349 12741 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:00.852722 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 232583us
I20260504 20:01:00.769364 14286 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:00.753345 14339 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.760195 14309 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260504 20:01:00.806298 12820 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 3
I20260504 20:01:00.868441 12790 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:00.871218 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 58602.6us
I20260504 20:01:00.859256 14187 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:00.900806 14187 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:00.879238 14294 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:00.906095 14370 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260504 20:01:01.023388 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 137140us
I20260504 20:01:00.864776 14320 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:01.036212 14348 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.912s user 0.021s sys 0.000s
I20260504 20:01:01.039127 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:01.080439 14348 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.030248 14344 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:01.113427 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 235069us
I20260504 20:01:01.087738 14259 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.026285 14391 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.097s
I20260504 20:01:01.126374 14207 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:01.138661 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:01.159094 14390 log.cc:927] Time spent T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260504 20:01:01.149592 14386 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.161365 14386 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.120751 14366 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:01.139326 14365 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.172267 14366 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.199059 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:01.200049 14266 consensus_meta.cc:287] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.804s user 0.015s sys 0.000s
I20260504 20:01:01.221086 14366 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.220876 14266 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:01.221736 14146 consensus_meta.cc:287] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.674s user 0.005s sys 0.009s
I20260504 20:01:01.222414 14146 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:01.123945 14344 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.127367 14392 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:01.225028 14392 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.225414 14344 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:01.209753 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:01.228446 14392 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:01.139695 14406 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.222718 13524 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:01.229599 14406 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.230338 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.213608 14365 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.205332 14373 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:01.218022 14376 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:01.235508 14266 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:01.260290 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.244683 14187 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:01.255285 14352 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.603s user 0.011s sys 0.005s
W20260504 20:01:01.263181 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 364408us
I20260504 20:01:01.249159 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:01.255846 14339 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:01.259110 14406 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:01.276136 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 134697us
I20260504 20:01:01.282030 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.280216 14351 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:01.286368 14351 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.288498 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1303) took 1018 ms. Trace:
I20260504 20:01:01.289203 14351 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:01.289330 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.268826 14320 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:01.290213 12749 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:01.291312 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:01.291435 14405 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.292163 12749 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:01.292048 14405 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.293236 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:01.294857 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:01.295083 14365 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.295611 12747 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 4
I20260504 20:01:01.271026 14352 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:01.306537 12890 consensus_meta.cc:287] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.632s user 0.008s sys 0.007s
I20260504 20:01:01.307375 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:01.307173 12890 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.308391 14380 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:01.309100 14380 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.310638 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:01.311998 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:01.308084 12899 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 3
I20260504 20:01:01.288890 12742 rpcz_store.cc:276] 0504 20:01:00.270360 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:00.270509 (+ 149us) service_pool.cc:224] Handling call
0504 20:01:00.337294 (+ 66785us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b5800000f98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:00.337508 (+ 214us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:00.337949 (+ 441us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:00.337983 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:00.338539 (+ 556us) log.cc:844] Serialized 26 byte log entry
0504 20:01:00.513852 (+175313us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:00.542381 (+ 28529us) log.cc:844] Serialized 14 byte log entry
0504 20:01:01.288040 (+745659us) spinlock_profiling.cc:243] Waited 4.99 ms on lock 0x7b18003bc020. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f015c55585c 00007f01598f3c0b 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f015990da95 00007f015c9a6440 00007f015c9b8af6 00007f015c9c9667
0504 20:01:01.288176 (+ 136us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:01.288253 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:01.288297 (+ 44us) raft_consensus.cc:1685] finished
0504 20:01:01.288312 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:01.288434 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20752,"prepare.queue_time_us":34043,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":8985,"raft.queue_time_us":109504,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":34000,"spinlock_wait_cycles":23138560,"thread_start_us":240990,"threads_started":2,"wal-append.queue_time_us":338294}
I20260504 20:01:01.312937 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.315299 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:01.305107 14355 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:01.320207 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:01.321339 12745 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:01.325163 12748 raft_consensus.cc:3055] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:01.334049 12748 raft_consensus.cc:740] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:01.332178 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:01.333041 12750 raft_consensus.cc:2393] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:01.337333 13499 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:01.252249 14373 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.331449 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:01.340515 12823 raft_consensus.cc:1240] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:01.308023 14352 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.284608 14386 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:01.363248 14361 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:01.365291 14348 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.349642 12748 consensus_queue.cc:260] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.394649 12748 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:01.369818 14051 consensus_queue.cc:1059] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:01.389099 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:01.397707 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:01.406744 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:01.385246 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:01.409759 13533 raft_consensus.cc:2410] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260504 20:01:01.412308 13499 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:01.387564 14406 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.422574 14406 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.391822 12890 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:01.391635 14361 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:01.421217 12747 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:01.451004 14361 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:01.455310 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:01.394212 14365 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:01.439559 14380 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.441068 14405 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:01.395526 14386 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:01.475222 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:01.478899 13525 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:01.441787 14406 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:01.482081 14411 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260504 20:01:01.442879 14372 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:01.443385 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:01.385397 14355 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.390893 14408 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:01.540925 14408 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.436872 14373 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.468916 13499 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:01.545300 14408 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:01.559369 14355 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.564232 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:01.567093 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:01.568928 13526 raft_consensus.cc:2368] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:01.470976 12892 raft_consensus.cc:1240] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:01.475198 12741 consensus_meta.cc:287] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.607s user 0.013s sys 0.004s
I20260504 20:01:01.573343 12741 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:01.573606 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1522) took 1093 ms. Trace:
I20260504 20:01:01.574142 12741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1305) took 1192 ms. Trace:
I20260504 20:01:01.574222 12892 rpcz_store.cc:276] 0504 20:01:00.480453 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:00.481959 (+ 1506us) service_pool.cc:224] Handling call
0504 20:01:01.470515 (+988556us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3a3e 000000000035a3c9 00007f016190b8e0 00007f01619d4784 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:01.470719 (+ 204us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:01.573222 (+102503us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:01.573479 (+ 257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2504448}
I20260504 20:01:01.574741 12741 rpcz_store.cc:276] 0504 20:01:00.382244 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:00.823007 (+440763us) service_pool.cc:224] Handling call
0504 20:01:01.574064 (+751057us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:01.493472 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1525) took 1012 ms. Trace:
I20260504 20:01:01.494362 12748 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
W20260504 20:01:01.499089 14360 consensus_meta.cc:287] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.713s user 0.017s sys 0.000s
I20260504 20:01:01.501263 14365 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.576043 12898 rpcz_store.cc:276] 0504 20:01:00.481335 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:00.481481 (+ 146us) service_pool.cc:224] Handling call
0504 20:01:00.482143 (+ 662us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:00.482628 (+ 485us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:00.482665 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:00.501846 (+ 19181us) log.cc:844] Serialized 487 byte log entry
0504 20:01:00.637294 (+135448us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:00.675569 (+ 38275us) log.cc:844] Serialized 14 byte log entry
0504 20:01:01.486099 (+810530us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54002943a0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c929b13 00007f015c9af19f 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc
0504 20:01:01.486160 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:01.486255 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:01.486303 (+ 48us) raft_consensus.cc:1685] finished
0504 20:01:01.493241 (+ 6938us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:01.493402 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":80460,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":4266,"spinlock_wait_cycles":375051008,"thread_start_us":252232,"threads_started":2,"wal-append.queue_time_us":198358}
I20260504 20:01:01.579279 14365 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.576749 14360 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.582795 14360 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.624572 12647 catalog_manager.cc:5671] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 3, leader changed from ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:01.648573 12648 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39332 (request call id 221) took 2177 ms. Trace:
I20260504 20:01:01.649305 12648 rpcz_store.cc:276] 0504 20:00:59.470389 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:00:59.471125 (+ 736us) service_pool.cc:224] Handling call
0504 20:00:59.609051 (+137926us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540030f054. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c918470 00007f015c9b731a 00007f016275a46c 00007f01628543ee 00007f0160d3a4a4 00007f0160d3a432 00007f0160d3a361 00007f0160d3a2dc 00007f0160d39592 00007f015ace8bcc 00007f015ace8006
0504 20:01:01.648112 (+2039061us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:00.360685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:00.412534 (+ 51849us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:00.412582 (+ 48us) write_op.cc:435] Acquired schema lock
0504 20:01:00.412603 (+ 21us) tablet.cc:662] Decoding operations
0504 20:01:00.413282 (+ 679us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:00.413335 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:00.413364 (+ 29us) tablet.cc:685] Acquiring locks for 4 operations
0504 20:01:00.413766 (+ 402us) tablet.cc:701] Row locks acquired
0504 20:01:00.413787 (+ 21us) write_op.cc:260] PREPARE: finished
0504 20:01:00.413977 (+ 190us) write_op.cc:270] Start()
0504 20:01:00.414097 (+ 120us) write_op.cc:276] Timestamp: P: 1777924860413924 usec, L: 0
0504 20:01:00.414120 (+ 23us) op_driver.cc:348] REPLICATION: starting
0504 20:01:00.414802 (+ 682us) log.cc:844] Serialized 1677 byte log entry
0504 20:01:00.704916 (+290114us) op_driver.cc:464] REPLICATION: finished
0504 20:01:01.146519 (+441603us) write_op.cc:301] APPLY: starting
0504 20:01:01.152718 (+ 6199us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:01.194825 (+ 42107us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:01.208069 (+ 13244us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:01.213718 (+ 5649us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:01.213990 (+ 272us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0504 20:01:01.214088 (+ 98us) write_op.cc:312] APPLY: finished
0504 20:01:01.214387 (+ 299us) log.cc:844] Serialized 41 byte log entry
0504 20:01:01.214612 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:01.214741 (+ 129us) write_op.cc:454] Released schema lock
0504 20:01:01.214811 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:01.215264 (+ 453us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:01.215318 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"spinlock_wait_cycles":293302528,"child_traces":[["op",{"apply.queue_time_us":412320,"apply.run_cpu_time_us":3669,"apply.run_wall_time_us":73797,"num_ops":4,"prepare.queue_time_us":178535,"prepare.run_cpu_time_us":3650,"prepare.run_wall_time_us":73612,"replication_time_us":269521,"thread_start_us":247844,"threads_started":2}]]}
I20260504 20:01:01.434834 13524 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260504 20:01:01.579604 12820 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.693s user 0.010s sys 0.006s
I20260504 20:01:01.627646 12647 catalog_manager.cc:5671] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:01.693563 12647 catalog_manager.cc:5671] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:01.515049 12899 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:01.674079 14376 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:01.677368 14302 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:01.685340 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 216510us
I20260504 20:01:01.685496 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:01.689126 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 385175us
I20260504 20:01:01.679304 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:01.695706 12647 catalog_manager.cc:5671] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: config changed from index 3 to 4, VOTER 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) evicted. New cstate: current_term: 1 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:01.709007 13533 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:01.711694 14381 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.713091 13499 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:01.712577 14381 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.714893 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.716845 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:01.721692 14259 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.690151 14348 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:01.722815 14410 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.690316 14344 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:01.736129 14398 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:01.697947 14339 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.726073 14183 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.731755 14388 raft_consensus.cc:3055] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
W20260504 20:01:01.732633 12635 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 91ff767ed7c24b97940a459421881055 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 82787d8e64d24ca885ca517aa8ed1be2 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:01.691150 14352 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.733999 13499 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:01.702615 14376 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:01.723085 14259 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.691890 14355 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:01.736440 14344 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.752611 14384 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:01.752918 14339 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.754101 14183 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.720311 14302 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.697927 12820 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
W20260504 20:01:01.803634 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 69728.2us
I20260504 20:01:01.737525 14347 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.743192 14225 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.756181 14388 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:01.732657 14309 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.077s
I20260504 20:01:01.814309 14382 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:01.737087 14221 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.822695 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:01.799657 14364 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:01.800411 14182 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:01.805682 14065 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:01.755777 14361 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.808913 14380 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:01.808328 14146 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:01.822805 14384 raft_consensus.cc:697] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:01.842612 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.800191 14355 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:01.795981 14352 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.826941 14094 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.736931 14398 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.743883 14410 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.826260 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1471) took 1129 ms. Trace:
I20260504 20:01:01.824467 14225 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.884407 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:01.899888 12899 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:01.880479 13528 raft_consensus.cc:2368] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:01.820278 14347 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.901186 13499 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:01.853374 14099 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:01.855738 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.946882 12746 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:01.855598 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:01.950234 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:01.882268 14339 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.958073 12743 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:01:01.883239 14259 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:01.886349 14418 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20260504 20:01:02.012427 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:01.878526 14302 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.958938 12822 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:01.883589 14381 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.005044 12743 raft_consensus.cc:606] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:01.845742 14182 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.011442 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:01.844211 14221 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.011590 14281 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:01.879603 14388 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.949209 14389 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:02.082549 14389 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.083029 14389 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:02.083619 14398 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.084561 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1401) took 1420 ms. Trace:
I20260504 20:01:01.847752 14065 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.085615 14292 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:02.086238 12791 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:02.087397 14065 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.087455 12791 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:02.087996 14292 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:01.935951 14225 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.088640 14292 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:01.936137 14347 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.089108 12791 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:02.046520 14385 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:02.090034 14360 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:02.090499 14360 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.090250 14385 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.092835 14385 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:01.905711 14294 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:02.093845 14294 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.084970 12897 rpcz_store.cc:276] 0504 20:01:00.664460 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:00.665665 (+ 1205us) service_pool.cc:224] Handling call
0504 20:01:00.679462 (+ 13797us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:00.854346 (+174884us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:00.854401 (+ 55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:01.300900 (+446499us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b2c0005d1f8. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015b2a8c81 00007f015b2b85e2 00007f015c9e9de1 00007f015c9ae81a 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22
0504 20:01:01.301362 (+ 462us) log.cc:844] Serialized 26 byte log entry
0504 20:01:01.679582 (+378220us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:01.679634 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:01.679714 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:02.084222 (+404508us) raft_consensus.cc:1685] finished
0504 20:01:02.084261 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:02.084490 (+ 229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":84547,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":41150,"spinlock_wait_cycles":2872576,"thread_start_us":552037,"threads_started":2,"wal-append.queue_time_us":378766}
I20260504 20:01:02.094743 12816 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:02.095788 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.095988 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.096491 14294 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:02.096645 12746 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:02.097286 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:01.868520 14380 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.098412 14281 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:01.999760 14396 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.100041 14380 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.100488 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.058305 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.101107 12745 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:02.101243 14281 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.101375 12821 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:02.102069 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.102656 12745 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:02.103828 12747 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260504 20:01:02.007226 14417 log.cc:927] Time spent T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260504 20:01:02.104988 14352 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:01.867128 14146 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.100093 14396 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.108422 14146 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.109119 14396 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.109637 13497 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:02.110944 14259 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:02.113170 14344 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.114094 14302 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.114370 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:02.114722 14302 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:02.058892 12865 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.715s user 0.022s sys 0.051s
I20260504 20:01:02.058185 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.116955 14183 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.117295 12822 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:02.068928 12743 raft_consensus.cc:614] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:02.073773 12823 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:02.071962 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:01.898845 12820 rpcz_store.cc:276] 0504 20:01:00.690016 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:00.693195 (+ 3179us) service_pool.cc:224] Handling call
0504 20:01:01.815071 (+1121876us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:02.006143 14335 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:01.898175 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:02.070925 12741 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 2: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:02.123171 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.113229 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:02.126658 12746 raft_consensus.cc:1275] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:02.087726 14291 raft_consensus.cc:3055] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:02.090927 14360 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:02.055702 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:02.139837 12791 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:02.140367 12825 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:01.957981 14099 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.097865 14355 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.114995 14259 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:02.118276 12819 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
W20260504 20:01:02.119184 12865 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.776s user 0.023s sys 0.051s
I20260504 20:01:01.884673 14384 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:01.897518 14094 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:02.068542 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:02.072762 14381 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:02.121871 12742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.006s sys 0.028s
W20260504 20:01:02.159087 12742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s user 0.007s sys 0.028s
I20260504 20:01:02.125108 13526 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:02.122215 12894 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.008s sys 0.026s
W20260504 20:01:02.159854 12894 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s user 0.008s sys 0.026s
I20260504 20:01:02.079996 14361 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.136197 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.141117 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:02.059369 12863 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.042s sys 0.056s
I20260504 20:01:02.140954 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:02.163637 12791 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:02.154804 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 811477us
W20260504 20:01:02.137710 14349 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.020s sys 0.026s
I20260504 20:01:02.137418 14302 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:02.166953 12863 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.822s user 0.042s sys 0.056s
W20260504 20:01:02.178265 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 839566us
I20260504 20:01:02.167953 12818 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:02.174578 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.175171 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.180001 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:02.180594 12745 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 4
I20260504 20:01:02.181058 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.175453 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.165901 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:02.139102 14291 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:02.182765 14291 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.184250 14291 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:02.177829 12746 raft_consensus.cc:2393] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 2: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:02.187989 14365 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:02.188951 14221 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.189685 14423 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:02.190492 14423 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:02.191705 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.146591 14366 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:02.194160 14392 consensus_queue.cc:1048] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:02.177753 14407 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260504 20:01:02.177459 14349 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s user 0.020s sys 0.026s
I20260504 20:01:02.195609 14407 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:02.196185 14407 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:02.197355 12865 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:02.197768 14415 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:02.199208 14410 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.177975 14381 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.202126 14182 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.194108 14366 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } attrs { replace: true } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.202570 14381 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.206214 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:02.206436 12714 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:02.206914 12819 raft_consensus.cc:606] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:02.154336 14309 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:02.221220 12818 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:02.209894 12714 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:02.175971 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.210392 12865 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:02.229532 13497 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:02.223948 14366 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:02.232084 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.177285 14367 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:02.233886 14367 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.234431 14367 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:02.121454 14335 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.238312 12863 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:02.209214 14339 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:02.239084 12714 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:02.176635 14183 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.244326 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.249657 12750 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:02.246706 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.254894 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.255810 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.256683 12819 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:02.226197 14309 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.242933 14361 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:02.224157 14349 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.264988 12714 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:02.260864 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.265483 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:02.266263 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:02.236398 14229 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:02.258368 14281 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:02.249907 14182 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:02.340793 14351 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:02.248160 14344 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:02.270929 12819 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:02.342157 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.244226 14339 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.248298 14381 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260504 20:01:02.360209 12715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.009s sys 0.019s
W20260504 20:01:02.391390 12715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s user 0.010s sys 0.020s
W20260504 20:01:02.391793 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 621499us
I20260504 20:01:02.359997 12745 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:02.363469 12818 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
W20260504 20:01:02.363516 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 99358.2us
W20260504 20:01:02.365341 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 90924.3us
I20260504 20:01:02.271880 14309 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:02.387588 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.259661 14183 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.390141 14347 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.450814 14347 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.368234 14182 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.447252 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.468152 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.446345 12819 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:02.381141 14229 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.393563 14351 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.378563 14221 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260504 20:01:02.451480 14421 log.cc:927] Time spent T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.384s user 0.001s sys 0.001s
I20260504 20:01:02.265350 14396 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:02.447278 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.453430 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.502727 12816 raft_consensus.cc:2393] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:02.461707 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:02.452400 14423 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:02.464386 14392 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:02.464840 14347 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.392019 14339 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:02.368876 14349 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.454792 14344 raft_consensus.cc:697] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:02.391319 14381 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.342242 14389 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.495414 14182 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:02.503888 14183 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.513590 12715 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:02.511639 14392 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.513747 14396 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.529120 14392 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:02.530608 14389 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.526110 14051 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:02.531836 14051 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:02.532898 13533 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260504 20:01:02.497004 14221 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:02.499166 14388 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:02.512315 14428 log.cc:927] Time spent T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20260504 20:01:02.530805 14396 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.520817 14381 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:02.521854 14099 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:02.522459 14408 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:02.578099 12894 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1527) took 1275 ms. Trace:
I20260504 20:01:02.583858 12715 leader_election.cc:400] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:02.598912 12715 leader_election.cc:403] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:02.517310 13499 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:02.614831 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 97619.7us
I20260504 20:01:02.620682 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:02.621541 12898 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:02.541195 14221 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:02.532322 14051 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.630940 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.635557 12899 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
W20260504 20:01:02.631876 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 44305.2us
I20260504 20:01:02.542390 14094 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.637928 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.638288 12716 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:02.579473 14415 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:02.638854 12893 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:01:02.648582 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 123764us
I20260504 20:01:02.648928 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.649885 12899 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:02.650434 12715 leader_election.cc:400] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:02.650904 12715 leader_election.cc:403] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:02.653038 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.653927 12899 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:02.656891 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.658891 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:02.659541 12897 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:02.511381 12735 tablet_service.cc:1558] Processing DeleteTablet for tablet 593a993b4ff341c9b98629e2c051ce00 with delete_type TABLET_DATA_TOMBSTONED (TS 82787d8e64d24ca885ca517aa8ed1be2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57450
I20260504 20:01:02.662259 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.664094 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.670336 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:02.680204 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.682987 12899 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:02.683328 12891 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:02.683898 13533 raft_consensus.cc:1240] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:02.665729 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.596190 12894 rpcz_store.cc:276] 0504 20:01:01.302973 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:01.328921 (+ 25948us) service_pool.cc:224] Handling call
0504 20:01:01.329337 (+ 416us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:01.329662 (+ 325us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:01.329690 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:01.330230 (+ 540us) log.cc:844] Serialized 26 byte log entry
0504 20:01:02.160225 (+829995us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:02.577118 (+416893us) log.cc:844] Serialized 14 byte log entry
0504 20:01:02.577437 (+ 319us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:02.577545 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:02.577610 (+ 65us) raft_consensus.cc:1685] finished
0504 20:01:02.577630 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:02.577995 (+ 365us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58183,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":13766,"thread_start_us":1149748,"threads_started":2,"wal-append.queue_time_us":899733}
I20260504 20:01:02.688125 12892 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:02.701117 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.630188 14335 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:02.630746 14389 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:02.702500 14398 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:02.708289 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.709096 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.749351 12895 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:02.633069 14410 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:02.633307 14423 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:02.713672 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.702723 14424 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:02.703775 14426 log.cc:927] Time spent T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260504 20:01:02.737258 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:02.738435 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:02.711956 14437 log.cc:927] Time spent T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260504 20:01:02.746982 14065 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.751487 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.752151 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:02.784672 12819 raft_consensus.cc:606] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:02.706457 14420 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:02.754912 12896 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:02.661024 14146 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:02.711021 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.638983 14408 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:02.756181 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 490737us
I20260504 20:01:02.712399 14183 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:02.713218 14416 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:02.763469 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 117) took 1436 ms. Trace:
I20260504 20:01:02.695257 14347 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:02.770179 14396 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:02.774119 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.749922 14335 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:02.805338 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.807140 12744 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:02.808724 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.762555 14378 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:02.787690 14291 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.603s user 0.016s sys 0.000s
I20260504 20:01:02.714498 14344 consensus_queue.cc:237] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.765568 14410 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.845902 12790 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:02.818397 12742 rpcz_store.cc:276] 0504 20:01:01.327104 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:01.327290 (+ 186us) service_pool.cc:224] Handling call
0504 20:01:01.347834 (+ 20544us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:01.369477 (+ 21643us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:01.385343 (+ 15866us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:01.386725 (+ 1382us) log.cc:844] Serialized 52 byte log entry
0504 20:01:02.159547 (+772822us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:02.159587 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:02.159658 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:02.763143 (+603485us) raft_consensus.cc:1685] finished
0504 20:01:02.763176 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:02.763382 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9477,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":21622,"thread_start_us":772239,"threads_started":1,"wal-append.queue_time_us":827016}
I20260504 20:01:02.819710 12715 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:02.816205 12817 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.511s user 0.008s sys 0.001s
I20260504 20:01:02.768604 14409 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.758823 14389 raft_consensus.cc:697] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:02.711549 14438 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260504 20:01:02.784185 14229 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.838944 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.840096 12747 raft_consensus.cc:2393] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:02.931841 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.932530 12747 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:02.933399 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.935251 12747 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:02.936451 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:02.937907 12824 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.049s sys 0.044s
I20260504 20:01:02.840813 12891 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
W20260504 20:01:02.836910 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:02.848824 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:02.947117 12824 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.580s user 0.050s sys 0.044s
I20260504 20:01:02.815563 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.854312 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:02.819803 12898 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
W20260504 20:01:02.913004 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 50151.9us
W20260504 20:01:02.922472 13526 consensus_meta.cc:287] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.713s user 0.006s sys 0.006s
I20260504 20:01:02.913874 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.949060 12819 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:02.912791 14420 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:02.918174 14366 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:02.805848 14443 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.770592 14423 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.771559 14424 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.922762 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.987556 12894 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:02.756944 14398 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:02.938035 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.994200 12747 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:01:02.830327 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:02.939908 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:02.823432 14349 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:02.810007 14431 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:02.841723 14183 raft_consensus.cc:697] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:02.913702 12821 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:02.949987 12895 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: [2.5-3.6]
I20260504 20:01:02.955570 12714 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:03.020227 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 64782.9us
I20260504 20:01:02.951771 14351 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:02.917057 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 71587.6us
I20260504 20:01:03.030416 14366 consensus_queue.cc:1059] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:01:02.836908 14416 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:02.951423 14433 log.cc:927] Time spent T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.307s user 0.000s sys 0.001s
I20260504 20:01:02.851500 14335 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.858273 14378 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:02.853906 14347 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:03.034703 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 315003us
I20260504 20:01:02.995901 12896 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:03.036895 12899 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:03.019428 12821 raft_consensus.cc:606] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:02.794413 14065 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.039043 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260504 20:01:03.040256 12750 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:02.856777 14396 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.028062 12892 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:03.028825 12893 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:02.981316 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:03.065567 14146 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:03.066177 14146 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:03.066563 14146 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:03.067277 14441 log.cc:927] Time spent T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.338s user 0.000s sys 0.000s
W20260504 20:01:03.069455 14436 log.cc:927] Time spent T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.358s user 0.001s sys 0.000s
I20260504 20:01:02.959204 13526 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:02.965274 14051 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:02.931219 14409 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:02.993371 12791 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.652s (SENT) [suppressed 54 similar messages]
W20260504 20:01:03.097358 14437 log.cc:927] Time spent T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260504 20:01:03.027542 14431 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.114794 13526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40940 (request call id 193) took 1279 ms. Trace:
I20260504 20:01:03.120234 13526 rpcz_store.cc:276] 0504 20:01:01.831643 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:01.836665 (+ 5022us) service_pool.cc:224] Handling call
0504 20:01:03.098993 (+1262328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:03.072155 12863 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:02.994416 14229 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:02.990940 14422 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:03.079123 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:03.143879 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 120) took 1193 ms. Trace:
I20260504 20:01:03.133989 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.091261 14435 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:03.139791 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:03.124516 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 52613us
I20260504 20:01:03.151614 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:01:03.144349 12817 rpcz_store.cc:276] 0504 20:01:01.950003 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:01.950129 (+ 126us) service_pool.cc:224] Handling call
0504 20:01:01.950531 (+ 402us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:02.000247 (+ 49716us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:02.000282 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:02.008802 (+ 8520us) log.cc:844] Serialized 26 byte log entry
0504 20:01:02.237464 (+228662us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:02.926788 (+689324us) log.cc:844] Serialized 14 byte log entry
0504 20:01:02.929710 (+ 2922us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:02.929888 (+ 178us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:03.143607 (+213719us) raft_consensus.cc:1685] finished
0504 20:01:03.143636 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:03.143801 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49375,"prepare.queue_time_us":212149,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":52210,"raft.run_cpu_time_us":6419,"raft.run_wall_time_us":59764,"thread_start_us":769127,"threads_started":2,"wal-append.queue_time_us":538832}
I20260504 20:01:03.160709 12863 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:03.160931 12743 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.007s sys 0.013s
W20260504 20:01:03.161952 12743 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s user 0.008s sys 0.013s
W20260504 20:01:03.175858 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 182483us
I20260504 20:01:03.183995 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.185688 13499 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:03.186204 14445 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:03.187036 14445 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.191010 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.183823 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.196755 12821 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:03.203413 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 115) took 1263 ms. Trace:
I20260504 20:01:03.204782 12824 rpcz_store.cc:276] 0504 20:01:01.939960 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:01.940118 (+ 158us) service_pool.cc:224] Handling call
0504 20:01:01.947361 (+ 7243us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:01.947741 (+ 380us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:01.947771 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:02.007432 (+ 59661us) log.cc:844] Serialized 26 byte log entry
0504 20:01:02.263465 (+256033us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:02.947613 (+684148us) log.cc:844] Serialized 14 byte log entry
0504 20:01:03.202938 (+255325us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:03.203047 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:03.203111 (+ 64us) raft_consensus.cc:1685] finished
0504 20:01:03.203128 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:03.203321 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":289646,"prepare.queue_time_us":186402,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":31,"raft.queue_time_us":267121,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":9544,"thread_start_us":769469,"threads_started":2,"wal-append.queue_time_us":466982}
I20260504 20:01:03.210762 14445 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.128476 14366 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:02.994812 14443 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.138206 14384 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:03.142900 14351 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:03.230312 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.232892 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.145169 14463 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20260504 20:01:03.097915 14389 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.142544 14229 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.180799 14435 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.158680 14422 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.267154 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:03.258919 12748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.009s sys 0.041s
I20260504 20:01:03.292043 14398 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.285768 14431 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.286090 14444 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:03.261623 12741 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:03.259039 14434 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:03.227833 14065 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:03.488375 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 307850us
W20260504 20:01:03.594550 14376 consensus_meta.cc:287] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.795s user 0.017s sys 0.004s
I20260504 20:01:03.290763 14408 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:03.259903 14453 log.cc:927] Time spent T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
W20260504 20:01:03.690037 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 455743us
I20260504 20:01:03.326695 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:03.502605 14451 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:03.433593 12897 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.774s user 0.016s sys 0.004s
I20260504 20:01:03.695281 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1477) took 1442 ms. Trace:
W20260504 20:01:03.591347 14344 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.006s sys 0.012s
W20260504 20:01:03.690083 12748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.224s user 0.025s sys 0.054s
I20260504 20:01:03.333146 14302 raft_consensus.cc:3055] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:03.697091 14302 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:03.692680 14434 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.695755 12746 rpcz_store.cc:276] 0504 20:01:02.252825 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.253014 (+ 189us) service_pool.cc:224] Handling call
0504 20:01:02.253421 (+ 407us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:02.259067 (+ 5646us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:02.259095 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:02.259560 (+ 465us) log.cc:844] Serialized 26 byte log entry
0504 20:01:02.750710 (+491150us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:02.750754 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:02.750834 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:03.695044 (+944210us) raft_consensus.cc:1685] finished
0504 20:01:03.695072 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:03.695222 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":404464,"threads_started":1,"wal-append.queue_time_us":632763}
W20260504 20:01:03.335642 14455 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20260504 20:01:03.226608 14351 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:03.695456 14451 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.701632 14351 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:03.702334 14229 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:03.702587 14434 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.262040 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:03.286080 14367 consensus_meta.cc:287] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.051s user 0.019s sys 0.013s
I20260504 20:01:03.703639 14367 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.417857 14286 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:03.704774 14286 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:03.329298 14392 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.800s user 0.010s sys 0.000s
I20260504 20:01:03.331385 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:03.290360 14378 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.706539 14392 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:03.324558 14309 consensus_meta.cc:287] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.868s user 0.010s sys 0.000s
I20260504 20:01:03.707445 14286 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.707660 14309 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.228806 14348 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:03.549871 14355 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:03.710279 12821 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:03.238240 14384 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.321257 14464 tablet_replica.cc:333] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: stopping tablet replica
I20260504 20:01:03.705860 14451 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:03.713091 12863 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:03.713249 13524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 418) took 1039 ms. Trace:
I20260504 20:01:03.713994 14094 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:03.713373 14309 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.353760 12894 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:03.713958 13524 rpcz_store.cc:276] 0504 20:01:02.674286 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.702969 (+ 28683us) service_pool.cc:224] Handling call
0504 20:01:03.345215 (+642246us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:03.345454 (+ 239us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:03.712155 (+366701us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:03.712196 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:03.712248 (+ 52us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:03.712285 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:03.712981 (+ 696us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54004b0fd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:03.713041 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:03.713180 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":870514688}
I20260504 20:01:03.716390 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:03.332098 14407 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.135s user 0.012s sys 0.002s
I20260504 20:01:03.276472 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:03.720439 12896 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 2: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:03.720587 14443 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:03.225211 14366 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.244076 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.721782 12816 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:01:03.723057 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 242576us
I20260504 20:01:03.461033 14448 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:03.724092 12714 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:03.724208 14448 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.724875 14467 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:03.725291 14467 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:03.725652 14467 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:03.726097 14420 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:03.726326 12894 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:03.726549 14448 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:03.726754 14420 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:03.290009 14449 log.cc:927] Time spent T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.235s user 0.000s sys 0.001s
W20260504 20:01:03.711306 14344 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.008s sys 0.013s
I20260504 20:01:03.714576 14094 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:03.235726 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:03.729950 14360 consensus_meta.cc:287] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.572s user 0.026s sys 0.007s
I20260504 20:01:03.276278 14183 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.730283 14094 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:03.730427 14360 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:03.730818 14381 consensus_meta.cc:287] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.106s user 0.011s sys 0.021s
I20260504 20:01:03.731628 12822 raft_consensus.cc:2393] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:03.732084 14381 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.240772 14335 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:03.330440 14409 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:03.693493 14376 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.691622 14444 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:03.732460 14360 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.719655 14384 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.764534 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:03.764716 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 71512.4us
W20260504 20:01:03.713549 14259 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.014s sys 0.037s
I20260504 20:01:03.695453 12897 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:03.710186 14355 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.698449 14302 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.708616 14392 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.770982 12742 raft_consensus.cc:2393] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:03.713891 14367 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:03.771975 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:03.286674 14416 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:03.778964 14408 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:03.779681 14388 consensus_meta.cc:287] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.236s user 0.009s sys 0.010s
I20260504 20:01:03.779640 14408 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.757025 14444 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:03.782796 14221 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.141s user 0.011s sys 0.008s
I20260504 20:01:03.783442 14302 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:03.733472 14349 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:03.776054 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1531) took 1141 ms. Trace:
I20260504 20:01:03.785374 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1474) took 1558 ms. Trace:
I20260504 20:01:03.785777 12897 rpcz_store.cc:276] 0504 20:01:02.629858 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.630041 (+ 183us) service_pool.cc:224] Handling call
0504 20:01:03.770821 (+1140780us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:03.786355 12748 rpcz_store.cc:276] 0504 20:01:02.226783 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.227801 (+ 1018us) service_pool.cc:224] Handling call
0504 20:01:02.245219 (+ 17418us) spinlock_profiling.cc:243] Waited 9.67 ms on lock 0x7b5800000f98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:02.245464 (+ 245us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:02.465364 (+219900us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:02.465401 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:02.465931 (+ 530us) log.cc:844] Serialized 26 byte log entry
0504 20:01:03.696416 (+1230485us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:03.696462 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:03.700407 (+ 3945us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b5400243454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:03.700458 (+ 51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:03.785150 (+ 84692us) raft_consensus.cc:1685] finished
0504 20:01:03.785177 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:03.785311 (+ 134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14301,"raft.queue_time_us":14387,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":8126,"spinlock_wait_cycles":496697600,"thread_start_us":1230013,"threads_started":1,"wal-append.queue_time_us":1314295}
I20260504 20:01:03.783219 14221 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.789011 12818 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:03.786366 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.793623 14431 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:03.793937 14051 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:03.794443 14051 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.794353 14431 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:03.773396 14259 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.015s sys 0.037s
W20260504 20:01:03.776057 14182 consensus_meta.cc:287] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.250s user 0.010s sys 0.005s
I20260504 20:01:03.717736 14407 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:03.801116 14146 consensus_meta.cc:287] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.734s user 0.016s sys 0.006s
W20260504 20:01:03.802106 14339 consensus_meta.cc:287] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.281s user 0.014s sys 0.001s
W20260504 20:01:03.801175 14281 consensus_meta.cc:287] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.420s user 0.017s sys 0.002s
I20260504 20:01:03.816038 14355 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:03.816160 14281 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:03.816507 14366 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.817112 12891 raft_consensus.cc:1275] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:03.817365 14420 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.808079 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.818275 12817 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:03.819118 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.806396 12791 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:03.820665 14366 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:03.821029 14366 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:03.821408 14366 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:03.819258 12823 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:03.791229 14360 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:03.788712 14438 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:03.815485 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.828756 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.831275 12817 raft_consensus.cc:2368] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:03.831991 14431 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.832239 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:03.832516 12823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1492) took 1657 ms (client timeout 1682 ms). Trace:
W20260504 20:01:03.832923 12823 rpcz_store.cc:269] 0504 20:01:02.174763 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.180796 (+ 6033us) service_pool.cc:224] Handling call
0504 20:01:03.832337 (+1651541us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b540015e5d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:03.832453 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3765876096}
I20260504 20:01:03.803610 14349 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.841542 12714 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:03.809065 14182 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.845602 14182 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:03.845913 14478 log.cc:927] Time spent T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.149s user 0.003s sys 0.001s
I20260504 20:01:03.850881 12863 leader_election.cc:400] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:03.851289 12863 leader_election.cc:403] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:03.823776 14146 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:03.869786 14457 log.cc:927] Time spent T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260504 20:01:03.869689 14065 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:03.868285 14335 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 3
I20260504 20:01:03.875568 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 128) took 1648 ms. Trace:
I20260504 20:01:03.830849 14438 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:03.902348 14470 log.cc:927] Time spent T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20260504 20:01:03.876032 12825 rpcz_store.cc:276] 0504 20:01:02.227458 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.230859 (+ 3401us) service_pool.cc:224] Handling call
0504 20:01:02.231325 (+ 466us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:02.534736 (+303411us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:02.534769 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:02.546648 (+ 11879us) log.cc:844] Serialized 52 byte log entry
0504 20:01:03.005893 (+459245us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:03.005960 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:03.425914 (+419954us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b5400180e54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:03.425977 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:03.875156 (+449179us) raft_consensus.cc:1685] finished
0504 20:01:03.875182 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:03.875484 (+ 302us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8144,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":41894,"spinlock_wait_cycles":678943488,"thread_start_us":368468,"threads_started":1,"wal-append.queue_time_us":1079570}
I20260504 20:01:03.849112 14431 consensus_queue.cc:1059] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260504 20:01:03.905634 14431 raft_consensus.cc:3055] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:03.906064 14431 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:03.907003 14431 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:03.908416 14431 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:03.869176 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:03.910125 12817 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 3
I20260504 20:01:03.910712 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1475) took 1683 ms. Trace:
I20260504 20:01:03.891678 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:03.911976 12823 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:03.893308 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:03.875568 14065 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:03.904901 14408 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:03.826710 14339 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.911253 12743 rpcz_store.cc:276] 0504 20:01:02.227217 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.241859 (+ 14642us) service_pool.cc:224] Handling call
0504 20:01:02.242247 (+ 388us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:02.245939 (+ 3692us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:02.245994 (+ 55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:02.249023 (+ 3029us) log.cc:844] Serialized 26 byte log entry
0504 20:01:03.162359 (+913336us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:03.162407 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:03.162481 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:03.910382 (+747901us) raft_consensus.cc:1685] finished
0504 20:01:03.910416 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:03.910624 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":83382,"raft.queue_time_us":2064,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":16176,"thread_start_us":777259,"threads_started":1,"wal-append.queue_time_us":1403538}
I20260504 20:01:03.938099 12749 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:03.943814 14051 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:03.947082 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1322) took 1131 ms. Trace:
I20260504 20:01:03.947690 12742 rpcz_store.cc:276] 0504 20:01:02.816036 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.862525 (+ 46489us) service_pool.cc:224] Handling call
0504 20:01:03.946782 (+1084257us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b5400012a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:03.946953 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1872574336}
I20260504 20:01:03.950316 14408 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:03.950896 14408 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:03.958453 14347 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:03.964486 14410 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:03.965559 14422 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:03.930519 14065 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:03.970635 14423 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:03.970526 14424 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:03.969290 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:03.974231 14376 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:03.978760 14464 raft_consensus.cc:2243] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260504 20:01:03.985668 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 116293us
I20260504 20:01:03.986461 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1420) took 1028 ms. Trace:
I20260504 20:01:03.986994 12891 rpcz_store.cc:276] 0504 20:01:02.957732 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.963096 (+ 5364us) service_pool.cc:224] Handling call
0504 20:01:03.458757 (+495661us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:03.459003 (+ 246us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:03.817506 (+358503us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:03.986132 (+168626us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b5400208fd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:03.986374 (+ 242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":861208832}
W20260504 20:01:03.997916 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 302947us
I20260504 20:01:03.998536 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:03.998991 14476 log.cc:927] Time spent T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.301s user 0.001s sys 0.000s
I20260504 20:01:03.999473 13532 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:04.005056 12864 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:03.935456 14407 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.006589 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.007450 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.008046 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.008767 14463 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:04.010659 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:04.011540 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.011824 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.012459 13528 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:04.014298 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:03.982230 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1490) took 1808 ms. Trace:
I20260504 20:01:04.016779 12822 rpcz_store.cc:276] 0504 20:01:02.174143 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.174271 (+ 128us) service_pool.cc:224] Handling call
0504 20:01:03.982023 (+1807752us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540014c8d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:03.982162 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3568531328}
I20260504 20:01:03.954792 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:04.017866 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1486) took 1165 ms. Trace:
I20260504 20:01:04.018307 12744 rpcz_store.cc:276] 0504 20:01:02.852343 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.963910 (+111567us) service_pool.cc:224] Handling call
0504 20:01:02.964300 (+ 390us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:02.964552 (+ 252us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:02.964575 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:03.127502 (+162927us) log.cc:844] Serialized 26 byte log entry
0504 20:01:03.428054 (+300552us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:03.428094 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:03.428202 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:03.998349 (+570147us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:04.010082 (+ 11733us) raft_consensus.cc:1685] finished
0504 20:01:04.010118 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:04.017771 (+ 7653us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106152832,"thread_start_us":211557,"threads_started":1,"wal-append.queue_time_us":552238}
I20260504 20:01:04.009971 14348 raft_consensus.cc:697] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:04.019982 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:04.018803 14221 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.020866 12823 raft_consensus.cc:2393] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:04.021180 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1315) took 1217 ms. Trace:
I20260504 20:01:04.045488 12749 rpcz_store.cc:276] 0504 20:01:02.804162 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.808374 (+ 4212us) service_pool.cc:224] Handling call
0504 20:01:04.020956 (+1212582us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540022f1d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:04.021111 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2509717376}
I20260504 20:01:04.021502 12819 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:04.020905 14348 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:04.023041 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.025424 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:04.050559 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:04.050889 12825 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 3
I20260504 20:01:04.027245 12647 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39316 (request call id 223) took 3963 ms. Trace:
I20260504 20:01:04.028087 14146 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.027860 14409 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:04.016430 12790 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:04.065001 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 48657.3us
I20260504 20:01:04.064760 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.031481 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.036675 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.067001 12743 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:04.037076 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.068104 12748 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:04.044358 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.058234 12715 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:04.067873 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:04.027819 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.071429 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:04.052047 12647 rpcz_store.cc:276] 0504 20:01:00.063593 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:00.066907 (+ 3314us) service_pool.cc:224] Handling call
0504 20:01:04.027156 (+3960249us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:02.188436 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:02.189050 (+ 614us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:02.189097 (+ 47us) write_op.cc:435] Acquired schema lock
0504 20:01:02.189118 (+ 21us) tablet.cc:662] Decoding operations
0504 20:01:02.189791 (+ 673us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:02.189909 (+ 118us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:02.189935 (+ 26us) tablet.cc:685] Acquiring locks for 4 operations
0504 20:01:02.190355 (+ 420us) tablet.cc:701] Row locks acquired
0504 20:01:02.190374 (+ 19us) write_op.cc:260] PREPARE: finished
0504 20:01:02.190516 (+ 142us) write_op.cc:270] Start()
0504 20:01:02.190637 (+ 121us) write_op.cc:276] Timestamp: P: 1777924862190475 usec, L: 0
0504 20:01:02.190659 (+ 22us) op_driver.cc:348] REPLICATION: starting
0504 20:01:02.191449 (+ 790us) log.cc:844] Serialized 1662 byte log entry
0504 20:01:02.794958 (+603509us) op_driver.cc:464] REPLICATION: finished
0504 20:01:03.666492 (+871534us) write_op.cc:301] APPLY: starting
0504 20:01:03.666639 (+ 147us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:03.666852 (+ 213us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:03.667012 (+ 160us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:03.667664 (+ 652us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:03.667930 (+ 266us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0504 20:01:03.667997 (+ 67us) write_op.cc:312] APPLY: finished
0504 20:01:03.678455 (+ 10458us) log.cc:844] Serialized 41 byte log entry
0504 20:01:03.811108 (+132653us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:03.824419 (+ 13311us) write_op.cc:454] Released schema lock
0504 20:01:03.826969 (+ 2550us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:03.867397 (+ 40428us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:03.868165 (+ 768us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18033,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":833447,"apply.run_cpu_time_us":19333,"apply.run_wall_time_us":192788,"num_ops":4,"prepare.queue_time_us":366270,"prepare.run_cpu_time_us":3381,"prepare.run_wall_time_us":10192,"replication_time_us":567053,"thread_start_us":724133,"threads_started":3,"wal-append.queue_time_us":150941}]]}
I20260504 20:01:04.075745 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:04.078354 12744 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:04.079309 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:04.079888 12748 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:04.080431 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.083161 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.080207 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:04.085099 12749 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:04.027745 14467 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.043450 14339 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.070969 14407 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:04.069011 14464 raft_consensus.cc:2272] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 20:01:04.075246 14420 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:04.100051 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:04.101197 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:04.111631 12821 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:04.096304 12825 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:04.114820 12714 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:04.111425 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.116241 12744 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:04.117218 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:04.122620 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.123625 12749 raft_consensus.cc:2393] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:04.123507 12748 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:04.099493 14381 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.129912 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:04.134446 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:04.135133 12825 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:04.100440 14376 raft_consensus.cc:476] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Not starting pre-election -- already a leader
I20260504 20:01:04.111074 14420 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:04.146934 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:04.149158 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:04.112730 14408 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.126041 14349 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:04.160071 14438 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:04.161844 14051 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:04.162323 14467 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.126932 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:04.174085 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:04.136900 14183 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:04.140844 14435 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.194775 12744 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:04.162786 14281 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:04.156942 14420 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:04.240248 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:04.194773 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 95329.7us
I20260504 20:01:04.196939 12821 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:04.249486 12825 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:04.247762 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.146265 14361 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260504 20:01:04.240154 12817 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:04.254734 12714 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:04.255265 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:04.155675 14410 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:04.258563 12714 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:04.262826 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:04.258931 14435 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
W20260504 20:01:04.260279 14094 consensus_meta.cc:287] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.529s user 0.018s sys 0.005s
I20260504 20:01:04.260850 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.193540 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.281240 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.265858 13499 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:04.162350 14051 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:04.283903 12744 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:04.240589 14183 raft_consensus.cc:697] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:04.271479 14438 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:04.273092 14444 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.277489 14489 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:04.283129 13499 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:04.269640 12825 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:04.290904 12897 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:04.306059 12791 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:04.263967 14462 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:04.316459 12898 raft_consensus.cc:2368] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:04.320499 12714 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:04.280642 14396 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:04.330646 12791 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:04.330808 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 47823.2us
I20260504 20:01:04.256494 14281 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:04.284161 14434 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:04.290186 14051 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:04.333132 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:04.335764 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.282328 14094 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.410351 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.411253 12891 raft_consensus.cc:1813] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Handling vote request from an unknown peer 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:04.412010 12891 raft_consensus.cc:2410] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260504 20:01:04.377436 14183 consensus_queue.cc:237] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:04.379565 14366 consensus_meta.cc:287] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.558s user 0.009s sys 0.010s
W20260504 20:01:04.380142 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 48466us
I20260504 20:01:04.376362 12806 tablet_service.cc:1558] Processing DeleteTablet for tablet dfa79aedaea6499facdfc14a3e7467f8 with delete_type TABLET_DATA_TOMBSTONED (TS 5ff0ef72ec3b4667893a0420d3c7f923 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53094
I20260504 20:01:04.426267 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.426555 14366 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.427253 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.427482 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.428141 12892 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:04.428990 12899 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:04.290447 14435 raft_consensus.cc:697] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:01:04.406600 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 52470.4us
I20260504 20:01:04.335578 14461 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:04.416360 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 83309.2us
I20260504 20:01:04.439474 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:04.441602 12791 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:04.337988 14483 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:04.335482 14438 raft_consensus.cc:697] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:04.337729 14229 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260504 20:01:04.383332 14485 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260504 20:01:04.423312 14309 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260504 20:01:04.424835 14475 log.cc:927] Time spent T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.712s user 0.005s sys 0.000s
I20260504 20:01:04.425417 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:04.406117 14351 consensus_meta.cc:287] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.685s user 0.016s sys 0.004s
W20260504 20:01:04.422298 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 101928us
I20260504 20:01:04.386229 14467 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:04.449975 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.450791 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.387424 14381 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:04.472041 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:04.473232 12817 raft_consensus.cc:2393] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:04.331503 14462 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:04.282127 14410 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:04.455683 12791 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:04.462340 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:04.478149 12893 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:04.478395 12791 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:04.469182 12825 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260504 20:01:04.479465 14408 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:04.474750 12899 raft_consensus.cc:1275] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:04.452158 14367 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:04.456485 14498 tablet_replica.cc:333] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: stopping tablet replica
I20260504 20:01:04.451681 14483 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:04.488015 14483 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:04.481210 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:04.458607 14376 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:04.454905 14347 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:04.474988 14444 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:04.485822 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:04.505991 12892 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:04.458439 14229 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:04.463079 14463 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:04.494738 14435 consensus_queue.cc:237] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.482941 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.501788 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57450 (request call id 576) took 1154 ms. Trace:
I20260504 20:01:04.501359 14438 consensus_queue.cc:237] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.474452 14351 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.483843 14309 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:04.481298 14381 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.487167 14467 raft_consensus.cc:697] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:04.601114 12899 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:04.490200 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.600981 12745 rpcz_store.cc:276] 0504 20:01:03.347767 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:03.372646 (+ 24879us) service_pool.cc:224] Handling call
0504 20:01:04.501547 (+1128901us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b5400243954. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b4b15 00007f015c9acdda 00007f0161982b89 00007f015aef1db4 00007f015aef1d42 00007f015aef1c71 00007f015aef1bec 00007f015aef0ea2 00007f015ace8bcc 00007f015ace8006
0504 20:01:04.501726 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297263488}
I20260504 20:01:04.484032 14366 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.622866 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:04.503815 14425 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:04.729990 12897 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 2: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:04.730180 14425 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:04.730791 14425 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:04.634440 14483 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.513873 14420 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:04.491506 14182 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.135s
I20260504 20:01:04.516661 14229 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:04.506150 13526 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:04.528331 14376 raft_consensus.cc:697] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:04.620110 14475 log.cc:927] Time spent T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260504 20:01:04.730537 14467 consensus_queue.cc:237] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.742837 14483 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:04.749097 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.754844 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.761585 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:04.771411 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:04.507834 14347 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:04.807219 12896 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:04.811488 12899 raft_consensus.cc:2368] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
W20260504 20:01:04.758358 14389 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.025s sys 0.027s
I20260504 20:01:04.837317 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260504 20:01:04.819993 14389 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.109s user 0.028s sys 0.035s
W20260504 20:01:04.782055 14335 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.906s user 0.014s sys 0.007s
I20260504 20:01:04.776412 14221 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:04.861217 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.552839 14463 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.779019 14281 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:04.780305 14361 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:04.870051 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 88221.7us
I20260504 20:01:04.835590 14424 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:04.887638 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:04.863209 14094 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:04.870615 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.871779 14335 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:04.894451 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.895172 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.897416 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.893682 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.904201 12746 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:04.907470 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.909504 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.910223 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.910626 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.911234 12747 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:04.911280 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.914513 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.916276 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.929477 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:04.931782 14420 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:04.876086 14464 ts_tablet_manager.cc:1916] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260504 20:01:04.869524 14422 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260504 20:01:04.946769 12892 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:04.943022 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:04.947698 12895 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:01:04.944443 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:04.915408 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.954478 13533 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260504 20:01:04.960438 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:04.961540 12894 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:04.965616 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.902884 14410 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:04.876533 14221 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.975088 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.985587 12746 raft_consensus.cc:1240] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:04.898080 14462 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:04.896922 14496 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:04.890444 14424 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.883697 14361 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:05.009243 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:05.013638 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 190756us
I20260504 20:01:05.014729 12865 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:05.018888 12865 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:04.969314 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:04.903339 14507 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:05.024504 12865 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:05.027712 12865 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:05.004974 14351 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:05.023568 14410 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:04.901692 14376 consensus_queue.cc:237] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:04.986709 14420 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260504 20:01:05.026058 12890 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:05.025938 14335 raft_consensus.cc:2764] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260504 20:01:05.030844 14281 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:05.155337 12898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1427) took 1888 ms (client timeout 1735 ms). Trace:
I20260504 20:01:05.246810 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:05.252398 12817 raft_consensus.cc:2393] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:05.251984 12898 rpcz_store.cc:269] 0504 20:01:03.266559 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:03.266821 (+ 262us) service_pool.cc:224] Handling call
0504 20:01:05.154991 (+1888170us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b54000f4e54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:05.155224 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1440803200}
I20260504 20:01:05.007854 14496 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:05.127995 14410 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:05.049357 14367 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:05.035642 14507 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:05.145246 14498 raft_consensus.cc:2243] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 20:01:05.249984 14347 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:05.146759 14381 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:05.249655 14497 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:05.249353 14094 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:05.308889 14372 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260504 20:01:05.319559 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 294571us
I20260504 20:01:05.299960 14410 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:05.368506 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:05.369969 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:05.370649 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 338623us
I20260504 20:01:05.331815 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:05.356652 14051 consensus_meta.cc:287] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.954s user 0.013s sys 0.001s
I20260504 20:01:05.313446 14367 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:05.413086 12893 consensus_meta.cc:287] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.934s user 0.010s sys 0.005s
I20260504 20:01:05.455632 12893 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
W20260504 20:01:05.434687 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:05.332077 14497 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:05.449681 14361 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:05.451582 14431 consensus_meta.cc:287] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.543s user 0.006s sys 0.014s
I20260504 20:01:05.433173 14498 raft_consensus.cc:2272] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 20:01:05.371666 12865 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:05.456416 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1449) took 1005 ms. Trace:
I20260504 20:01:05.436962 14463 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:05.467960 12742 raft_consensus.cc:2410] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260504 20:01:05.441560 14051 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:05.444949 14221 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:05.463953 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 92186.9us
I20260504 20:01:05.464313 12893 rpcz_store.cc:276] 0504 20:01:04.450668 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.462001 (+ 11333us) service_pool.cc:224] Handling call
0504 20:01:05.456348 (+994347us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:05.507205 12865 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:05.508941 12865 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:05.512881 14463 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:05.512805 14513 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:05.519567 12750 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:05.514292 14513 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:05.525506 14367 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:05.522022 14463 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:05.529592 14507 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:05.531080 14051 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:05.503827 14520 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:05.535331 12865 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:05.543655 12865 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:05.509964 14424 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:05.531606 14065 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.564s user 0.016s sys 0.003s
I20260504 20:01:05.565165 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:05.527571 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:05.574126 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 242629us
I20260504 20:01:05.527217 14496 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:05.536932 14520 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:05.582885 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:05.593796 14498 ts_tablet_manager.cc:1916] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260504 20:01:05.611654 12742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57934 (request call id 130) took 1598 ms (client timeout 1978 ms). Trace:
I20260504 20:01:05.595968 14520 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:05.617339 12742 rpcz_store.cc:269] 0504 20:01:04.013090 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.013546 (+ 456us) service_pool.cc:224] Handling call
0504 20:01:05.611211 (+1597665us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b540022fbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:05.611571 (+ 360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3223849856}
I20260504 20:01:05.590643 14424 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:05.623862 14512 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:05.625476 14512 consensus_queue.cc:1048] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:05.576004 14065 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:05.596757 12890 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:01:05.596236 14523 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:05.606554 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:05.651413 12748 raft_consensus.cc:2368] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:05.652488 14513 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:05.663384 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:05.664311 12747 raft_consensus.cc:2368] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:05.667562 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:05.669703 12894 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1435) took 1334 ms. Trace:
I20260504 20:01:05.670418 12894 rpcz_store.cc:276] 0504 20:01:04.335258 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.335501 (+ 243us) service_pool.cc:224] Handling call
0504 20:01:05.669299 (+1333798us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b54001fb3d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:05.669562 (+ 263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1276285696}
I20260504 20:01:05.632527 14463 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:05.682642 12788 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.011s sys 0.028s
W20260504 20:01:05.683211 12788 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s user 0.011s sys 0.028s
I20260504 20:01:05.724521 12820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:05.682688 14463 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:05.818207 14497 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:05.821852 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:05.819073 14522 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:05.875080 14373 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:05.864722 14146 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:05.944011 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 400442us
I20260504 20:01:05.945029 12865 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:05.946316 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 342884us
I20260504 20:01:05.876658 14373 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:05.915835 14525 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:05.958590 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 388093us
I20260504 20:01:05.959234 12791 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:05.961405 12791 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:05.971791 14424 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:05.975176 14518 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:05.975368 14424 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:05.976866 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:05.976572 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 136) took 1100 ms. Trace:
I20260504 20:01:05.964900 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:05.982355 12825 rpcz_store.cc:276] 0504 20:01:04.875777 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.876002 (+ 225us) service_pool.cc:224] Handling call
0504 20:01:04.876398 (+ 396us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:04.876668 (+ 270us) raft_consensus.cc:1534] Early marking committed up to index 6
0504 20:01:04.983107 (+106439us) log.cc:844] Serialized 14 byte log entry
0504 20:01:05.962522 (+979415us) log.cc:844] Serialized 14 byte log entry
0504 20:01:05.975397 (+ 12875us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:05.975469 (+ 72us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:05.975521 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:05.976340 (+ 819us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:05.976503 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28245,"prepare.queue_time_us":148375,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":208305,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":289,"thread_start_us":1061885,"threads_started":4,"wal-append.queue_time_us":457833}
I20260504 20:01:05.957064 14065 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:05.902554 14522 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:05.964717 14523 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:05.960098 14463 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:05.950595 14146 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:05.982411 12820 raft_consensus.cc:606] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:05.982723 14360 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:05.962239 14467 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.002s sys 0.027s
I20260504 20:01:06.038057 14539 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:05.995280 14518 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:05.996244 14384 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:06.040148 12820 raft_consensus.cc:614] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260504 20:01:06.046350 14467 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.003s sys 0.027s
I20260504 20:01:05.963584 14525 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.023716 14523 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:06.081007 14373 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.082305 14523 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:06.046656 14065 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:06.036934 14529 consensus_queue.cc:1048] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
W20260504 20:01:06.111264 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 146417us
I20260504 20:01:06.046986 14539 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:06.115525 12865 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:06.117268 12865 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:06.118956 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:06.089869 14373 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:06.096285 14398 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:06.065025 14384 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:06.122438 14229 consensus_meta.cc:287] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.339s user 0.014s sys 0.004s
W20260504 20:01:06.117244 14376 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.019s sys 0.041s
I20260504 20:01:06.077983 14520 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.117761 14539 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:06.148579 14502 log.cc:927] Time spent T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 1.343s user 0.004s sys 0.013s
I20260504 20:01:06.113631 14065 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:06.146559 13526 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 430) took 1897 ms (client timeout 1616 ms). Trace:
I20260504 20:01:06.144822 14522 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:06.168040 14530 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.224s user 0.002s sys 0.000s
I20260504 20:01:06.126971 14398 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:06.166112 13526 rpcz_store.cc:269] 0504 20:01:04.249399 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.256198 (+ 6799us) service_pool.cc:224] Handling call
0504 20:01:06.146250 (+1890052us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b54006ee954. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:06.146434 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":505618816}
I20260504 20:01:06.135015 14424 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:06.163618 14344 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:06.164135 14534 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:06.154366 14538 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 20:01:06.158440 14376 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.020s sys 0.041s
I20260504 20:01:06.130355 14373 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:06.185184 14425 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.454s user 0.008s sys 0.013s
I20260504 20:01:06.155006 14146 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.195796 12743 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:06.170177 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:06.149840 14229 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.168108 14065 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:06.178429 14398 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.179157 14384 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.172925 14525 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:06.193934 14410 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.831s user 0.018s sys 0.000s
I20260504 20:01:06.196516 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1333) took 1289 ms. Trace:
I20260504 20:01:06.185385 14520 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:06.205340 12743 rpcz_store.cc:276] 0504 20:01:04.907496 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.935241 (+ 27745us) service_pool.cc:224] Handling call
0504 20:01:06.195323 (+1260082us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b5800000f98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:06.195574 (+ 251us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:06.196258 (+ 684us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:06.196456 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30248832}
W20260504 20:01:06.050887 12791 connection.cc:254] RPC call timeout handler was delayed by -1.0152838660000043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3064s for a response
I20260504 20:01:06.184441 14424 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:06.200052 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 149815us
I20260504 20:01:06.212531 12821 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
W20260504 20:01:06.211357 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 139193us
I20260504 20:01:06.195427 14425 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.216419 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.217355 12894 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:01:06.220188 14525 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:06.214677 14366 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:06.220908 14525 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.221206 14366 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.211338 14376 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:06.226188 14376 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.228971 14376 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.232641 14373 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.233194 14425 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.214428 14522 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:06.239058 13499 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:06.240370 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.245028 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.245770 12893 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:06.225380 14366 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.211499 14410 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.250332 13499 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:06.251104 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 132253us
I20260504 20:01:06.260319 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:06.261328 14424 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:06.261988 12865 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:06.264833 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:06.248721 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.246162 14549 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:06.262382 14384 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:06.293465 14384 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:06.246805 14522 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:06.265000 14424 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:06.309793 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 43545us
W20260504 20:01:06.310820 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 1.29912e+06us
I20260504 20:01:06.310720 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:06.315117 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:06.316290 12748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1505) took 2179 ms (client timeout 2171 ms). Trace:
W20260504 20:01:06.316790 12748 rpcz_store.cc:269] 0504 20:01:04.129080 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.129531 (+ 451us) service_pool.cc:224] Handling call
0504 20:01:06.297667 (+2168136us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400243e54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:06.308566 (+ 10899us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3487175424}
I20260504 20:01:06.317554 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:06.318028 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 46756us
I20260504 20:01:06.318269 12822 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:06.319095 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:06.319303 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:06.323633 14229 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.307792 14545 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:06.326854 14546 consensus_queue.cc:1059] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260504 20:01:06.324692 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:06.328225 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:06.328702 12819 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:06.329041 13528 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:01:06.329237 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:06.329841 12818 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:06.280408 14549 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:06.333963 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:06.310227 14384 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:06.339124 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:06.340092 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:06.342312 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:06.342054 12823 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:06.341537 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:06.324555 14515 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:06.345863 12865 leader_election.cc:400] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:06.350095 12865 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:06.351208 12818 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:06.346374 14545 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:06.331662 14520 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:06.353091 14410 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:06.353214 14522 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:06.341786 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:06.356745 14546 raft_consensus.cc:3055] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:06.359519 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:06.362843 12819 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:06.397727 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:06.363396 14505 log.cc:927] Time spent T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 1.588s user 0.007s sys 0.015s
W20260504 20:01:06.367897 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 45811us
I20260504 20:01:06.370020 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.400218 12893 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:06.401173 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:06.402595 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.371230 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.377269 14424 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:06.379570 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:06.379815 14555 consensus_queue.cc:1059] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:06.411317 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:06.380368 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.380959 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.376963 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.414324 12898 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:06.415244 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:06.378814 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.416268 12898 raft_consensus.cc:2393] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:06.416572 12896 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:06.378500 12790 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:06.379222 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.379101 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.417364 12892 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:06.338521 14549 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:06.397469 13528 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:06.407178 12899 raft_consensus.cc:2376] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3. Re-sending same reply.
I20260504 20:01:06.418011 14448 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:06.418649 14448 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:06.419260 14448 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:06.408033 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:06.423280 12743 raft_consensus.cc:2410] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260504 20:01:06.397226 14546 raft_consensus.cc:740] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:06.394872 14525 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.412604 14555 raft_consensus.cc:3055] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:06.415107 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.379197 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:06.413465 12895 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:06.429157 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:06.438014 12897 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:06.377236 14515 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:06.627986 12791 leader_election.cc:400] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:06.645854 12791 leader_election.cc:403] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260504 20:01:06.475505 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:06.646777 12897 raft_consensus.cc:2393] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:06.643455 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:06.443212 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1522) took 1081 ms. Trace:
I20260504 20:01:06.649617 12749 rpcz_store.cc:276] 0504 20:01:05.362067 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:05.375623 (+ 13556us) service_pool.cc:224] Handling call
0504 20:01:06.442396 (+1066773us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b5800000f98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:06.442663 (+ 267us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:06.442878 (+ 215us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:06.442902 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:06.442947 (+ 45us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:06.442988 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:06.443046 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:06.443141 (+ 95us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35689344}
I20260504 20:01:06.650517 12714 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:06.691591 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 63882.2us
I20260504 20:01:06.693377 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:06.716300 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 65841.7us
I20260504 20:01:06.722808 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:06.474669 14500 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 1.665s user 0.011s sys 0.003s
W20260504 20:01:06.439749 14309 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.628s user 0.036s sys 0.044s
I20260504 20:01:06.599613 14448 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:06.740949 14309 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.959s user 0.039s sys 0.045s
I20260504 20:01:06.741395 14309 raft_consensus.cc:697] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:06.741941 14525 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:06.742588 14525 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:06.743291 14525 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:06.742750 14309 consensus_queue.cc:237] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.462216 14539 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.746989 12891 raft_consensus.cc:2393] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:06.748522 14539 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:06.430995 14555 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:06.750336 14555 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.752116 14555 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:06.752372 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:06.437359 14464 ts_tablet_manager.cc:1929] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260504 20:01:06.755517 14464 log.cc:1199] T 593a993b4ff341c9b98629e2c051ce00 P 82787d8e64d24ca885ca517aa8ed1be2: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-0-root/wals/593a993b4ff341c9b98629e2c051ce00
I20260504 20:01:06.723847 14548 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:06.717612 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:06.761183 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 43599.5us
I20260504 20:01:06.440255 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:06.737912 12748 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:01:06.739766 14508 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:06.740236 14515 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:06.792266 12750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57934 (request call id 138) took 1931 ms. Trace:
I20260504 20:01:06.797468 12750 rpcz_store.cc:276] 0504 20:01:04.860529 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.860841 (+ 312us) service_pool.cc:224] Handling call
0504 20:01:06.792079 (+1931238us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b54002bd4d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:06.792211 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1492673920}
I20260504 20:01:06.798092 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:06.794050 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1483) took 1553 ms. Trace:
I20260504 20:01:06.789258 14146 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:06.800285 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:06.407864 14520 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:06.801041 12895 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:06.728034 14558 log.cc:927] Time spent T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.089s user 0.003s sys 0.000s
I20260504 20:01:06.798982 12821 rpcz_store.cc:276] 0504 20:01:05.241112 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:05.241302 (+ 190us) service_pool.cc:224] Handling call
0504 20:01:05.247834 (+ 6532us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:06.213135 (+965301us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:06.793136 (+580001us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b5400399254. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:06.793977 (+ 841us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2218073856}
I20260504 20:01:06.803052 12823 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:06.721575 14498 ts_tablet_manager.cc:1929] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:01:06.806262 12818 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260504 20:01:06.806829 12818 pending_rounds.cc:85] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 5
I20260504 20:01:06.811374 12818 pending_rounds.cc:107] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260504 20:01:06.814569 12818 raft_consensus.cc:2889] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:06.813875 12893 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:06.812184 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:06.814343 14498 log.cc:1199] T dfa79aedaea6499facdfc14a3e7467f8 P 5ff0ef72ec3b4667893a0420d3c7f923: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-1-root/wals/dfa79aedaea6499facdfc14a3e7467f8
I20260504 20:01:06.814577 14464 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57450 (request call id 573) took 4429 ms. Trace:
I20260504 20:01:06.832145 14498 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53094 (request call id 511) took 2580 ms. Trace:
I20260504 20:01:06.832630 14498 rpcz_store.cc:276] 0504 20:01:04.266056 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.332099 (+ 66043us) service_pool.cc:224] Handling call
0504 20:01:04.456092 (+123993us) ts_tablet_manager.cc:1151] Deleting tablet dfa79aedaea6499facdfc14a3e7467f8
0504 20:01:04.456162 (+ 70us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 20:01:06.233156 (+1776994us) tablet_metadata.cc:665] Metadata flushed
0504 20:01:06.639230 (+406074us) tablet_metadata.cc:665] Metadata flushed
0504 20:01:06.832027 (+192797us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":353402,"tablet-delete.queue_time_us":29955,"thread_start_us":29508,"threads_started":1}
I20260504 20:01:06.441184 14546 consensus_queue.cc:260] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:06.835763 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 186746us
I20260504 20:01:06.804098 14508 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:06.839231 14520 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:06.887208 14448 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:06.894578 14508 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:06.830570 14464 rpcz_store.cc:276] 0504 20:01:02.452542 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.501052 (+ 48510us) service_pool.cc:224] Handling call
0504 20:01:03.290971 (+789919us) ts_tablet_manager.cc:1151] Deleting tablet 593a993b4ff341c9b98629e2c051ce00
0504 20:01:03.291135 (+ 164us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 20:01:06.190398 (+2899263us) tablet_metadata.cc:665] Metadata flushed
0504 20:01:06.425296 (+234898us) tablet_metadata.cc:665] Metadata flushed
0504 20:01:06.791475 (+366179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":279790,"tablet-delete.queue_time_us":565901,"thread_start_us":225835,"threads_started":1}
I20260504 20:01:06.897389 14549 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:06.900259 14549 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:06.914636 14525 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:06.945897 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 182600us
I20260504 20:01:06.948477 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:06.966390 12714 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:06.967905 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:06.970192 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:06.981248 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:06.982970 12741 raft_consensus.cc:2393] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:06.985960 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:06.989099 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:06.990065 12750 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:07.001065 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:07.027818 12743 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:07.002766 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.030719 12816 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:07.004123 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.032516 12817 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:01:07.006270 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:07.018532 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:07.020539 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.034637 13499 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.035185 12893 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:07.021986 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.022399 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.037035 12823 raft_consensus.cc:2393] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 2: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:07.022017 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.038697 12890 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:07.039091 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:07.044569 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:07.046586 13525 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:07.048511 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:07.049662 13526 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:07.050834 14065 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:07.051344 14065 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:07.051795 14065 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:07.052508 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:07.054891 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:07.055349 14146 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:07.056126 14146 raft_consensus.cc:697] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:07.045703 13524 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:07.059093 14183 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:07.060134 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:07.061126 13525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 437) took 2109 ms. Trace:
I20260504 20:01:07.060218 14146 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.061581 13525 rpcz_store.cc:276] 0504 20:01:04.951865 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.961564 (+ 9699us) service_pool.cc:224] Handling call
0504 20:01:06.295663 (+1334099us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:06.322383 (+ 26720us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:07.047230 (+724847us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:07.060730 (+ 13500us) spinlock_profiling.cc:243] Waited 723 ms on lock 0x7b54004b3f54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:07.061036 (+ 306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1701650560}
I20260504 20:01:07.063041 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:07.063215 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.065655 14183 raft_consensus.cc:697] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:07.067930 14183 consensus_queue.cc:237] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:06.917508 14525 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:07.087204 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 44923.2us
I20260504 20:01:07.109479 14387 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:07.114912 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49576 (request call id 13) took 1388 ms. Trace:
I20260504 20:01:07.115571 12820 rpcz_store.cc:276] 0504 20:01:05.698191 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:05.714876 (+ 16685us) service_pool.cc:224] Handling call
0504 20:01:07.069560 (+1354684us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400180e54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9a8c33 00007f0161984cca 00007f015aefd934 00007f015aefd8c2 00007f015aefd7f1 00007f015aefd76c 00007f015aefca22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:07.086135 (+ 16575us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":403678976}
I20260504 20:01:07.121665 12790 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:07.123744 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1530) took 1092 ms. Trace:
I20260504 20:01:07.124322 12744 rpcz_store.cc:276] 0504 20:01:06.031621 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:06.031779 (+ 158us) service_pool.cc:224] Handling call
0504 20:01:06.032192 (+ 413us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:06.117045 (+ 84853us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:06.117090 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:06.117134 (+ 44us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:06.117168 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:07.123235 (+1006067us) spinlock_profiling.cc:243] Waited 84.6 ms on lock 0x7b54002bcfd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:07.123301 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:07.123524 (+ 223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194538368}
I20260504 20:01:07.126180 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:07.127288 12821 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:07.127722 14384 consensus_meta.cc:287] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.784s user 0.011s sys 0.003s
I20260504 20:01:07.128010 14387 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:07.128391 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.128340 14384 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.129065 12750 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:07.138984 12743 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:07.143474 14467 consensus_queue.cc:1059] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:07.143880 12821 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:07.144881 12821 raft_consensus.cc:606] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:07.145306 14467 raft_consensus.cc:3055] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
W20260504 20:01:07.147611 14501 log.cc:927] Time spent T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 2.395s user 0.019s sys 0.013s
I20260504 20:01:07.132370 14302 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:07.161748 12895 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:07.163091 13532 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 202) took 2828 ms. Trace:
I20260504 20:01:07.163552 13532 rpcz_store.cc:276] 0504 20:01:04.336764 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.336887 (+ 123us) service_pool.cc:224] Handling call
0504 20:01:04.404568 (+ 67681us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:04.404881 (+ 313us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:04.404921 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:04.419303 (+ 14382us) log.cc:844] Serialized 26 byte log entry
0504 20:01:04.526170 (+106867us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:04.526222 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:04.526317 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:07.146410 (+2620093us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:07.149381 (+ 2971us) raft_consensus.cc:1685] finished
0504 20:01:07.149410 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:07.162801 (+ 13391us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":244879104,"thread_start_us":61785,"threads_started":1,"wal-append.queue_time_us":300513}
I20260504 20:01:07.151374 14302 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:07.150681 14467 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:07.178725 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.179456 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.182070 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.184772 12821 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:07.173691 14302 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:07.191432 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 223742us
I20260504 20:01:07.197335 14347 consensus_queue.cc:1059] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:07.198983 14335 raft_consensus.cc:3055] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:07.208935 12817 raft_consensus.cc:2410] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260504 20:01:07.210088 12742 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260504 20:01:07.212908 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:07.262518 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.188642 12821 raft_consensus.cc:606] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:07.209993 14384 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:07.277726 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.281026 14410 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:07.282045 14410 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.283860 14544 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:07.284755 14410 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:07.287678 14546 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:07.289535 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:07.291931 14347 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:07.292840 14347 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.294351 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.295636 14347 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:07.296921 12633 catalog_manager.cc:5002] TS 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): tablet dfa79aedaea6499facdfc14a3e7467f8 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:01:07.298571 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:07.299526 12897 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:07.277731 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1525) took 1278 ms. Trace:
I20260504 20:01:07.302686 12746 rpcz_store.cc:276] 0504 20:01:05.999503 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:06.000797 (+ 1294us) service_pool.cc:224] Handling call
0504 20:01:06.001205 (+ 408us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:06.061908 (+ 60703us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:06.061960 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:06.062025 (+ 65us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:06.062080 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:07.259233 (+1197153us) spinlock_profiling.cc:243] Waited 60.4 ms on lock 0x7b54002a1054. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:07.261279 (+ 2046us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:07.277540 (+ 16261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":138908544}
I20260504 20:01:07.305269 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.310159 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.315582 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:07.315652 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:07.316521 14065 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.317231 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:07.317297 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:07.321094 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.322587 12816 raft_consensus.cc:1240] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:07.318097 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.318306 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:07.328469 12824 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:07.329463 14065 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:07.330119 13532 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:07.318804 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:07.264074 14438 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.758s user 0.014s sys 0.023s
W20260504 20:01:07.332161 14438 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.909s user 0.015s sys 0.029s
I20260504 20:01:07.332525 12863 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:07.212311 14467 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.199455 14335 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:07.340713 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:07.340209 13499 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.345130 14543 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:07.345019 14523 consensus_queue.cc:1059] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:07.346830 14444 raft_consensus.cc:3055] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:07.346774 14525 consensus_queue.cc:1059] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260504 20:01:07.347472 14444 raft_consensus.cc:740] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:07.358682 14523 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:07.359385 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:07.357388 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:07.362039 14410 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:07.363758 14410 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.362142 14422 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:07.365716 14467 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:07.366421 14422 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.367561 14494 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:07.368856 14422 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:07.370088 14523 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.342391 14335 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.371934 14410 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:07.379374 14335 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:07.376113 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:07.329875 14302 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.377912 12863 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:07.214360 12646 catalog_manager.cc:5671] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 4 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:07.374089 14523 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:07.228027 13524 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:07.403512 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.399909 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.409307 12897 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:07.417671 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.433881 13499 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:07.434293 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:07.438794 13499 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:07.439402 12749 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:01:07.440095 12749 raft_consensus.cc:606] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:07.440330 14387 raft_consensus.cc:2804] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:07.440584 12749 raft_consensus.cc:614] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:07.440718 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.441073 14513 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:07.442463 14513 raft_consensus.cc:697] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:07.443665 13528 raft_consensus.cc:1240] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:07.440766 14387 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:07.444952 14387 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:07.446221 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.448494 12891 raft_consensus.cc:2368] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:07.450506 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:07.453243 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 75516.4us
I20260504 20:01:07.455219 12714 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:07.434082 14523 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:07.458321 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:07.458639 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:07.459484 12890 raft_consensus.cc:2368] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:07.463016 14302 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:07.463829 14094 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:07.464094 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:07.464457 14094 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:07.458408 12895 raft_consensus.cc:2376] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4. Re-sending same reply.
W20260504 20:01:07.482560 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:07.488607 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.487056 12863 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:07.468286 14438 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:07.484705 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.505833 12824 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:07.506639 12714 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:07.497907 14523 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:07.443696 14513 consensus_queue.cc:237] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.517061 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:07.516783 12745 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:07.487735 14444 consensus_queue.cc:260] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.515795 12824 raft_consensus.cc:606] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
W20260504 20:01:07.537442 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 263689us
I20260504 20:01:07.523651 14438 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260504 20:01:07.542308 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 70258.4us
I20260504 20:01:07.528679 14525 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:07.550163 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.552668 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.552852 12743 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260504 20:01:07.525421 14146 consensus_queue.cc:1059] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260504 20:01:07.554302 12821 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:07.554322 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.556321 14094 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:07.559610 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:07.560262 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 167760us
I20260504 20:01:07.561059 12817 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:07.569023 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.572389 12895 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
W20260504 20:01:07.585637 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 222526us
I20260504 20:01:07.586858 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:07.588853 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.595913 12743 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:07.593577 14525 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260504 20:01:07.605454 14348 consensus_queue.cc:1059] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260504 20:01:07.607105 14348 raft_consensus.cc:3055] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:07.607623 14348 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:07.600085 14543 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:07.608889 14348 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.610694 12891 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
W20260504 20:01:07.599637 12791 leader_election.cc:336] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.884s (SENT)
I20260504 20:01:07.616062 12790 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:07.618870 12895 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:07.624882 12791 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.585951 14523 consensus_queue.cc:1059] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260504 20:01:07.611639 14182 raft_consensus.cc:3055] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:07.633765 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:07.641114 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:07.621152 14347 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:07.643488 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 53024.7us
I20260504 20:01:07.644366 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:07.624197 12646 catalog_manager.cc:5671] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:07.647234 13532 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:07.648695 14302 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:07.616200 14543 consensus_queue.cc:1059] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260504 20:01:07.637095 14348 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:07.631223 14182 raft_consensus.cc:740] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:07.668889 12891 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:07.650954 14347 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.672828 14376 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:07.674926 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:07.675179 14561 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260504 20:01:07.676803 14463 raft_consensus.cc:3055] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:07.677058 12820 raft_consensus.cc:1275] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:07.679819 14182 consensus_queue.cc:260] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:07.680325 12635 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 91ff767ed7c24b97940a459421881055 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260504 20:01:07.666904 14302 raft_consensus.cc:697] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:07.687386 14578 log.cc:927] Time spent T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260504 20:01:07.687867 14579 log.cc:927] Time spent T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.094s user 0.003s sys 0.000s
I20260504 20:01:07.690644 14444 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:07.691474 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:07.702185 14463 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:07.715791 12746 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
W20260504 20:01:07.728456 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 140755us
I20260504 20:01:07.725739 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:07.725939 12635 catalog_manager.cc:5002] TS 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007): tablet 593a993b4ff341c9b98629e2c051ce00 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:01:07.730418 12819 raft_consensus.cc:606] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:07.738533 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.736500 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.750226 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.751860 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:07.752875 12750 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:07.754356 14302 consensus_queue.cc:237] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.763948 14572 raft_consensus.cc:3055] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:07.764449 14572 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:07.766978 14182 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:07.775523 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:07.782321 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:07.783636 13532 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:07.778266 13499 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:07.725982 14525 consensus_queue.cc:1048] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260504 20:01:07.792443 12790 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:07.791076 12895 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:07.796118 14347 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:07.796707 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260504 20:01:07.798369 14463 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.799723 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:01:07.799921 14463 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:07.805677 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:07.789634 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:07.807585 12716 connection.cc:254] RPC call timeout handler was delayed by -2.9271573749999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2372s for a response
I20260504 20:01:07.765527 14572 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:07.822898 14589 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20260504 20:01:07.823319 14229 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:07.823863 14229 raft_consensus.cc:697] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:07.824877 14229 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.826388 14387 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.830437 14549 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:07.831180 14549 raft_consensus.cc:697] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:07.832492 14549 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:07.835031 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 56925us
W20260504 20:01:07.839386 14580 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.055s user 0.003s sys 0.001s
I20260504 20:01:07.811019 14546 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:07.842571 14572 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:07.842767 14546 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.848482 12897 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:07.852960 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:07.857565 12716 connection.cc:254] RPC call timeout handler was delayed by -2.800257375000001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2513s for a response
I20260504 20:01:07.852039 14546 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:07.868929 12716 connection.cc:254] RPC call timeout handler was delayed by -2.5688268700000663s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
W20260504 20:01:07.869525 12716 connection.cc:254] RPC call timeout handler was delayed by -2.5112268700000868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
I20260504 20:01:07.856901 13524 raft_consensus.cc:1240] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:07.872081 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:07.865592 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.863561 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:07.873530 12716 connection.cc:254] RPC call timeout handler was delayed by -2.4925455669999792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260504 20:01:07.879262 12716 connection.cc:254] RPC call timeout handler was delayed by -2.3269455670000525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260504 20:01:07.879736 12716 connection.cc:254] RPC call timeout handler was delayed by -2.3003814129999682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16720000000000002s for a response
W20260504 20:01:07.880149 12716 connection.cc:254] RPC call timeout handler was delayed by -2.21962687000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22450000000000003s for a response
W20260504 20:01:07.880528 12716 connection.cc:254] RPC call timeout handler was delayed by -2.1774573750000172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3205s for a response
I20260504 20:01:07.871286 14387 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:07.887547 12716 connection.cc:254] RPC call timeout handler was delayed by -2.13259752700003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20260504 20:01:07.888782 12716 connection.cc:254] RPC call timeout handler was delayed by -2.0496772330000113s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
W20260504 20:01:07.889227 12716 connection.cc:254] RPC call timeout handler was delayed by -1.962325858999975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260504 20:01:07.890117 12716 connection.cc:254] RPC call timeout handler was delayed by -1.9353772330000538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17880000000000001s for a response
W20260504 20:01:07.890472 12716 connection.cc:254] RPC call timeout handler was delayed by -1.8795772330000773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18500000000000003s for a response
W20260504 20:01:07.890784 12716 connection.cc:254] RPC call timeout handler was delayed by -1.8608096230000228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260504 20:01:07.891067 12716 connection.cc:254] RPC call timeout handler was delayed by -1.8111772330000804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
W20260504 20:01:07.891341 12716 connection.cc:254] RPC call timeout handler was delayed by -1.7474462999999787s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
W20260504 20:01:07.891609 12716 connection.cc:254] RPC call timeout handler was delayed by -1.7239896050000425s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
W20260504 20:01:07.891885 12716 connection.cc:254] RPC call timeout handler was delayed by -1.6626772330000676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
W20260504 20:01:07.892149 12716 connection.cc:254] RPC call timeout handler was delayed by -1.6340629610000406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16820000000000002s for a response
W20260504 20:01:07.892412 12716 connection.cc:254] RPC call timeout handler was delayed by -1.5924772330000678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2169s for a response
W20260504 20:01:07.892663 12716 connection.cc:254] RPC call timeout handler was delayed by -1.5321381769999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
W20260504 20:01:07.892830 12716 connection.cc:254] RPC call timeout handler was delayed by -1.5025650100000121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
W20260504 20:01:07.892997 12716 connection.cc:254] RPC call timeout handler was delayed by -1.243987028000106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2542s for a response
W20260504 20:01:07.893147 12716 connection.cc:254] RPC call timeout handler was delayed by -1.2198772330000338s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25830000000000003s for a response
I20260504 20:01:07.894848 12865 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.894737 14569 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:07.895824 14569 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:07.898245 14569 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:07.900547 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:07.878480 14546 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:07.908672 13528 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 2. Current term is 3. Ops: []
W20260504 20:01:07.909484 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:07.918792 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 115618us
I20260504 20:01:07.919579 12865 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.916232 12741 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:07.925948 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:07.926028 14347 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:07.930693 12892 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:07.911780 14335 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:07.936932 12791 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:07.937907 12865 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.938392 14347 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:07.938948 14376 consensus_queue.cc:1059] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260504 20:01:07.923575 14569 consensus_queue.cc:1059] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260504 20:01:07.945139 13532 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:07.948617 14444 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:07.949328 14569 consensus_queue.cc:1243] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer ef6057d9a49e4942b47ea0fcd66ab6f7 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260504 20:01:07.949939 14569 raft_consensus.cc:3055] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:07.950242 14569 raft_consensus.cc:740] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:07.951251 14569 consensus_queue.cc:260] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.922484 14546 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.959825 14546 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:07.961933 14376 raft_consensus.cc:3055] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:07.962518 14376 raft_consensus.cc:740] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:07.963697 14376 consensus_queue.cc:260] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:07.964324 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.965970 14519 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:07.966114 14376 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:07.966897 14519 raft_consensus.cc:697] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:07.966964 12865 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.970954 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:07.972149 14519 consensus_queue.cc:237] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:07.973199 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:07.969544 14335 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:07.979116 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:07.985410 12865 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:07.985504 14349 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:07.987110 14463 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:07.971053 14546 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:07.970315 14569 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:07.983654 14065 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.013s
I20260504 20:01:08.001147 12741 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:08.003060 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.007784 14438 raft_consensus.cc:2764] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260504 20:01:08.011325 12890 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:08.013945 14549 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:08.023387 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.025156 14335 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.027371 12865 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.043390 12716 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.045003 14146 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.048730 14546 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:08.049376 14546 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.052260 14546 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:08.052659 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:08.061189 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 82205.4us
I20260504 20:01:08.063164 14546 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260504 20:01:08.074031 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 46789.8us
I20260504 20:01:08.074913 12865 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.086246 14546 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.086627 12865 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.087486 14546 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.094821 12716 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.095986 12865 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.096817 14146 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.098312 12865 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.099848 12865 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.103180 12865 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.089176 14349 raft_consensus.cc:697] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:08.112526 12865 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.113379 14523 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:08.113842 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:08.108997 14347 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.115828 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.109043 14463 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.117086 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.101195 14546 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:08.118441 14546 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:08.118906 14546 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.116607 12741 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:08.120155 12892 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260504 20:01:08.118978 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.118466 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.122992 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.123340 12749 raft_consensus.cc:2393] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:08.123790 12892 raft_consensus.cc:2410] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260504 20:01:08.118811 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:08.124334 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.125100 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.125690 12892 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:01:08.126034 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:08.100911 14335 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.128955 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.130625 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.135794 13497 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.136160 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.137367 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.137785 14348 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:08.138319 14348 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:08.138715 14348 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:08.139106 14549 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:08.139564 14549 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:08.139955 14549 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.143510 14463 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:08.141475 14349 consensus_queue.cc:237] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.144204 14463 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.145521 14146 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:08.146009 14146 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:08.146394 14146 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.146494 14463 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:08.147953 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.148772 12893 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:08.148933 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.149207 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.150038 12898 raft_consensus.cc:2393] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:08.150139 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.154947 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:08.154099 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.157763 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.158404 12897 raft_consensus.cc:1275] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:08.158841 12716 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.159888 12898 raft_consensus.cc:1275] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:08.159991 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.160168 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.160770 14355 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:08.160810 12742 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:08.161495 14494 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:08.162477 14494 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.162485 12865 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.163579 14444 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.164319 12865 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.164841 14444 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:08.165100 14410 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:08.165302 14494 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:08.165510 12865 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.167176 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:08.169169 12741 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:08.169315 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:08.172435 12864 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.172564 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.165270 14444 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:08.173247 12895 raft_consensus.cc:2376] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3. Re-sending same reply.
I20260504 20:01:08.173545 14444 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.176358 14372 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:08.182325 14335 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.183380 12741 raft_consensus.cc:1240] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:08.183517 14463 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:08.182367 12865 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.192968 14347 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:08.194056 14347 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:08.193840 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.194456 14347 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.197438 14146 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.201871 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:08.202054 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.202541 12746 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:08.203047 12890 raft_consensus.cc:2376] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4. Re-sending same reply.
W20260504 20:01:08.209057 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 48913.7us
I20260504 20:01:08.210021 12716 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.211401 12716 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.212244 14355 raft_consensus.cc:2764] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
W20260504 20:01:08.211851 12791 leader_election.cc:336] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.592s (SENT)
I20260504 20:01:08.210971 14569 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:08.214187 14569 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:08.214111 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:08.217018 12791 leader_election.cc:336] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.648s (SENT)
I20260504 20:01:08.215502 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.218014 12898 raft_consensus.cc:2376] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3. Re-sending same reply.
I20260504 20:01:08.217597 14348 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.218919 14569 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.220440 13532 raft_consensus.cc:1240] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:08.201026 14549 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.224094 14549 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.186262 14546 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.225642 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.226014 14348 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.227211 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.227939 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.228336 14349 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:08.229009 14349 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.227964 14494 consensus_queue.cc:1059] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260504 20:01:08.231284 14410 raft_consensus.cc:3055] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:08.231721 14410 raft_consensus.cc:740] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:08.232607 14146 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.232769 14410 consensus_queue.cc:260] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.235126 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.239814 14410 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:08.244722 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.246538 12716 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.247373 14355 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.248296 14355 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:08.248821 14355 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:08.249230 12716 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.249466 14355 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.249938 14546 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:08.252636 14444 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.253942 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.254217 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:08.263731 12897 raft_consensus.cc:2376] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4. Re-sending same reply.
I20260504 20:01:08.263805 13532 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:08.256059 14347 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.265281 12790 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:08.254689 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.254294 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.256027 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.254684 14349 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:08.261396 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.269304 12899 raft_consensus.cc:2368] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:08.270509 12892 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:08.265576 12746 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:01:08.255321 12791 leader_election.cc:336] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.728s (SENT)
I20260504 20:01:08.272248 12791 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.273192 14349 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.271458 14438 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.273943 14604 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:08.274358 14604 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:08.274739 14604 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:08.276249 14410 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:08.276732 14410 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.271490 12890 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:08.280226 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:08.282095 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:08.282829 12741 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.287024 14444 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:08.287791 14335 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.288764 14410 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:08.289398 12895 raft_consensus.cc:1240] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:08.289779 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:08.290442 12745 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.289434 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.291640 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.287710 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:08.295215 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.297225 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:08.297850 12743 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.297947 14410 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:08.298520 14347 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:08.299399 13524 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:08.303095 14349 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:08.305191 14302 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:08.303877 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 54772.1us
I20260504 20:01:08.314113 12745 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:08.316037 12865 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.317147 14347 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:08.314601 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 59337.7us
I20260504 20:01:08.317734 14347 raft_consensus.cc:697] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:08.306226 14302 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:08.317979 12791 leader_election.cc:336] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.735s (SENT)
I20260504 20:01:08.318481 12791 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.319271 14349 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.318794 14347 consensus_queue.cc:237] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.324458 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:08.323491 14302 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:08.330108 12716 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:08.338156 12791 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.945s (SENT) [suppressed 32 similar messages]
I20260504 20:01:08.346830 12743 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:08.351269 12865 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.353513 14410 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:08.354183 14410 raft_consensus.cc:697] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:08.359803 14410 consensus_queue.cc:237] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.362169 14355 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.365154 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.365901 14065 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:08.366477 14065 raft_consensus.cc:697] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:08.367553 14065 consensus_queue.cc:237] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:08.368803 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 48322.9us
I20260504 20:01:08.370558 12716 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.400439 12716 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.371434 14146 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:08.413242 14438 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.411492 14146 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:08.414601 14146 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:08.415110 12716 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.415994 14182 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:08.426674 12716 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.427891 14355 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.410395 14604 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.422734 14525 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260504 20:01:08.444104 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.446848 12716 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.449404 14435 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.449553 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.450387 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:08.458895 12895 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:08.460489 14605 raft_consensus.cc:993] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:08.461050 14605 raft_consensus.cc:1081] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:08.464311 13532 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:08.466090 14348 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:01:08.466387 14519 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260504 20:01:08.468392 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:08.473227 14604 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:08.456015 14435 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:08.478328 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:08.480700 12817 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:08.480036 12716 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.486035 14569 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.486418 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:08.488164 12741 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:08.489514 14569 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.490569 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.490738 12865 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:08.493310 14347 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:08.490808 12646 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36606 (request call id 117) took 8400 ms. Trace:
I20260504 20:01:08.493866 14347 raft_consensus.cc:697] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:08.494098 14349 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.494940 14347 consensus_queue.cc:237] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.496654 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.493826 12646 rpcz_store.cc:276] 0504 20:01:00.089998 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:00.090221 (+ 223us) service_pool.cc:224] Handling call
0504 20:01:08.490721 (+8400500us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:07.926538 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:07.927218 (+ 680us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:07.927275 (+ 57us) write_op.cc:435] Acquired schema lock
0504 20:01:07.927306 (+ 31us) tablet.cc:662] Decoding operations
0504 20:01:07.927768 (+ 462us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:07.927849 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:07.927892 (+ 43us) tablet.cc:685] Acquiring locks for 2 operations
0504 20:01:07.977973 (+ 50081us) tablet.cc:701] Row locks acquired
0504 20:01:07.978016 (+ 43us) write_op.cc:260] PREPARE: finished
0504 20:01:07.978213 (+ 197us) write_op.cc:270] Start()
0504 20:01:07.978370 (+ 157us) write_op.cc:276] Timestamp: P: 1777924867978142 usec, L: 0
0504 20:01:07.978423 (+ 53us) op_driver.cc:348] REPLICATION: starting
0504 20:01:08.023103 (+ 44680us) log.cc:844] Serialized 907 byte log entry
0504 20:01:08.290116 (+267013us) op_driver.cc:464] REPLICATION: finished
0504 20:01:08.430107 (+139991us) write_op.cc:301] APPLY: starting
0504 20:01:08.430297 (+ 190us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:08.430482 (+ 185us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:08.439751 (+ 9269us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:08.475258 (+ 35507us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:08.475537 (+ 279us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 20:01:08.475601 (+ 64us) write_op.cc:312] APPLY: finished
0504 20:01:08.475897 (+ 296us) log.cc:844] Serialized 29 byte log entry
0504 20:01:08.476099 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:08.476198 (+ 99us) write_op.cc:454] Released schema lock
0504 20:01:08.476259 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:08.476700 (+ 441us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:08.476733 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":121415,"apply.run_cpu_time_us":2955,"apply.run_wall_time_us":55353,"num_ops":2,"prepare.queue_time_us":160729,"prepare.run_cpu_time_us":25845,"prepare.run_wall_time_us":259405,"raft.queue_time_us":93295,"raft.run_cpu_time_us":16627,"raft.run_wall_time_us":42429,"replication_time_us":310163,"spinlock_wait_cycles":1918848,"thread_start_us":278304,"threads_started":4,"wal-append.queue_time_us":125323}]]}
I20260504 20:01:08.479097 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.499343 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.503497 14349 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.506726 12645 catalog_manager.cc:5671] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:01:08.509258 12743 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:08.513562 14569 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:08.513509 14509 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:08.514238 14569 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:08.514899 14569 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:08.517154 14182 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.518603 12791 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.520601 12645 catalog_manager.cc:5671] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:08.523726 12645 catalog_manager.cc:5671] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:01:08.527005 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 47080.1us
I20260504 20:01:08.528782 14376 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:08.529251 14376 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:08.529884 14376 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:08.526831 12645 catalog_manager.cc:5671] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 5, leader changed from ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 5 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:08.529227 12648 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39338 (request call id 227) took 6278 ms. Trace:
I20260504 20:01:08.532286 12648 rpcz_store.cc:276] 0504 20:01:02.251143 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:02.263277 (+ 12134us) service_pool.cc:224] Handling call
0504 20:01:08.528956 (+6265679us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":10}
I20260504 20:01:08.533659 12716 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.534725 12645 catalog_manager.cc:5671] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:08.533078 14410 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:08.539678 14410 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.544435 12791 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.546389 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.548966 14435 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.555315 14348 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:08.556452 14348 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.559867 14410 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:08.564585 14229 raft_consensus.cc:993] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:08.566332 14229 raft_consensus.cc:993] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:08.566916 14604 raft_consensus.cc:1081] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:08.567344 14229 raft_consensus.cc:1081] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:08.570799 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:08.572407 14348 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.572717 12645 catalog_manager.cc:5671] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:01:08.573418 12864 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.577030 14410 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.578215 14146 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.581122 14146 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.582639 13532 raft_consensus.cc:1240] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:08.584007 14569 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.584450 14410 consensus_queue.cc:1059] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:08.574817 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.586383 14604 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:08.586833 14604 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:08.577023 12645 catalog_manager.cc:5671] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:08.587273 14604 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.585934 14410 raft_consensus.cc:3055] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:08.588290 14410 raft_consensus.cc:740] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:08.588768 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.575119 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.590217 12742 raft_consensus.cc:2368] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 3. Current term is 4.
W20260504 20:01:08.590426 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 58153.9us
I20260504 20:01:08.589510 14410 consensus_queue.cc:260] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.591653 13499 leader_election.cc:400] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 3. Current term is 4.
I20260504 20:01:08.592021 13499 leader_election.cc:403] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:08.588128 12645 catalog_manager.cc:5671] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:08.592342 12791 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:08.592579 14410 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.593017 14509 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.593159 14349 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:08.593539 12645 catalog_manager.cc:5671] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:01:08.594748 14349 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:08.595121 12645 catalog_manager.cc:5671] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:08.596704 12645 catalog_manager.cc:5671] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:01:08.599536 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.601161 14569 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:08.573691 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 46034.5us
I20260504 20:01:08.606478 12645 catalog_manager.cc:5671] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:01:08.584790 13524 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:08.616281 12716 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.621762 12790 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:08.622766 14229 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:08.623267 14229 raft_consensus.cc:697] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:08.624262 14229 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.627177 14569 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.630601 14376 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.632747 14376 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.608028 14509 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 3. Current term is 4.
I20260504 20:01:08.634971 12716 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.641074 14569 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.641570 14604 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.650321 12818 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:08.650602 12716 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.651575 14569 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.652858 12716 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.653057 14569 consensus_queue.cc:1059] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:01:08.653996 14435 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.654242 14569 raft_consensus.cc:3055] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:08.654676 14569 raft_consensus.cc:740] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:08.656080 14569 consensus_queue.cc:260] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.657550 14604 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:08.657673 14569 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.663015 12791 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:08.667361 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:08.663872 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.677425 14519 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:08.671485 12647 catalog_manager.cc:5671] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 3, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 3 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:08.678870 14525 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.679874 12791 leader_election.cc:400] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:08.682946 12791 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.683743 14302 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:08.684252 14302 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:08.684695 14302 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.696870 14302 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.698920 14604 consensus_queue.cc:1059] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:08.700923 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:08.701026 14569 consensus_queue.cc:1048] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:08.701715 12743 raft_consensus.cc:1240] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:08.701974 14604 raft_consensus.cc:3055] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:08.702967 14604 raft_consensus.cc:740] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:08.703644 14569 consensus_queue.cc:1048] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:01:08.703979 14604 consensus_queue.cc:260] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.703979 14302 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:08.705658 14604 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.706519 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.711673 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.717586 12742 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:08.718698 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.719903 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.719372 12892 raft_consensus.cc:2376] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2. Re-sending same reply.
I20260504 20:01:08.724881 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.725989 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.726444 12745 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:01:08.725680 12894 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:08.729600 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.730451 12743 raft_consensus.cc:2393] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:08.725667 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.735486 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.725450 12648 catalog_manager.cc:5671] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:08.742952 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.744994 12648 catalog_manager.cc:5671] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 3 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 4 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:08.746245 12891 raft_consensus.cc:2393] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:08.747962 12648 catalog_manager.cc:5671] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 2 to 3, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 3 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:08.749769 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.750571 12891 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:08.751827 12892 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37318
I20260504 20:01:08.744133 14544 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:08.752585 12898 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37318
I20260504 20:01:08.753317 12898 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:01:08.753794 12898 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.737007 12741 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:08.758795 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.743784 13497 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.763116 13497 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.752612 14544 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.764554 14549 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.765282 14549 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:08.765601 14544 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:08.766215 12892 raft_consensus.cc:476] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Not starting forced leader election -- already a leader
I20260504 20:01:08.768378 12743 raft_consensus.cc:1240] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:08.766897 12894 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:01:08.772289 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:08.772925 12743 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:08.767374 12899 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37318
I20260504 20:01:08.774241 13499 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:08.774325 12899 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:01:08.774946 12899 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:08.777774 14372 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:08.778242 14372 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:08.778575 14372 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:08.785149 13497 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.773564 12894 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:08.788823 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.793133 12898 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.803380 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:08.803793 12896 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:08.791049 12648 catalog_manager.cc:5671] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 3 to 5, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 5 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:08.804375 12893 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:08.808064 12745 raft_consensus.cc:1240] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:08.809926 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.753569 12890 raft_consensus.cc:1275] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:08.809511 12647 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39332 (request call id 222) took 4691 ms. Trace:
I20260504 20:01:08.813181 12743 raft_consensus.cc:1275] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260504 20:01:08.821547 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.822296 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.823145 12894 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.813174 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.814407 12898 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:08.815685 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.816058 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.826282 12817 raft_consensus.cc:3055] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:08.826646 12817 raft_consensus.cc:740] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:08.816619 12742 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:08.816423 12648 catalog_manager.cc:5671] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:08.804833 14544 raft_consensus.cc:2804] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:08.804160 14348 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.833659 12750 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:08.830058 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:08.817093 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:08.835863 12716 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.827391 12817 consensus_queue.cc:260] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.838315 12817 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 4
I20260504 20:01:08.863463 12750 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:08.865335 12746 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:08.869741 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 76225.4us
W20260504 20:01:08.880256 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:08.819931 12647 rpcz_store.cc:276] 0504 20:01:04.118083 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:04.126498 (+ 8415us) service_pool.cc:224] Handling call
0504 20:01:08.809446 (+4682948us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:08.695988 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:08.696548 (+ 560us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:08.696583 (+ 35us) write_op.cc:435] Acquired schema lock
0504 20:01:08.696595 (+ 12us) tablet.cc:662] Decoding operations
0504 20:01:08.696953 (+ 358us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:08.697007 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:08.697033 (+ 26us) tablet.cc:685] Acquiring locks for 1 operations
0504 20:01:08.697208 (+ 175us) tablet.cc:701] Row locks acquired
0504 20:01:08.697225 (+ 17us) write_op.cc:260] PREPARE: finished
0504 20:01:08.697366 (+ 141us) write_op.cc:270] Start()
0504 20:01:08.697470 (+ 104us) write_op.cc:276] Timestamp: P: 1777924868697319 usec, L: 0
0504 20:01:08.697485 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 20:01:08.698302 (+ 817us) log.cc:844] Serialized 530 byte log entry
0504 20:01:08.704715 (+ 6413us) op_driver.cc:464] REPLICATION: finished
0504 20:01:08.705169 (+ 454us) write_op.cc:301] APPLY: starting
0504 20:01:08.705252 (+ 83us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:08.705361 (+ 109us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:08.705467 (+ 106us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:08.705725 (+ 258us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:08.705906 (+ 181us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 20:01:08.705951 (+ 45us) write_op.cc:312] APPLY: finished
0504 20:01:08.706131 (+ 180us) log.cc:844] Serialized 23 byte log entry
0504 20:01:08.706300 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:08.706393 (+ 93us) write_op.cc:454] Released schema lock
0504 20:01:08.706431 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:08.706820 (+ 389us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:08.706849 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9035,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":260,"apply.run_cpu_time_us":1598,"apply.run_wall_time_us":1598,"num_ops":1,"prepare.queue_time_us":161,"prepare.run_cpu_time_us":2471,"prepare.run_wall_time_us":2611,"replication_time_us":7091}]]}
W20260504 20:01:08.888870 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 53086.6us
I20260504 20:01:08.833720 14347 consensus_queue.cc:1048] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260504 20:01:08.857157 12741 raft_consensus.cc:1240] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:08.892724 12819 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:08.863498 14438 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.850322 14544 raft_consensus.cc:697] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:08.902890 12891 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:08.904192 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:08.913169 12749 raft_consensus.cc:2393] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:08.913446 12896 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:08.906684 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:08.913300 14544 consensus_queue.cc:237] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:08.875486 12822 raft_consensus.cc:1275] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:08.890288 12817 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:08.916929 12863 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:08.920586 12825 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:08.895743 12716 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.940780 12894 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:08.942760 12893 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
W20260504 20:01:08.945963 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 58538.6us
W20260504 20:01:08.947477 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 55739.8us
I20260504 20:01:08.948670 12899 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:08.951519 12750 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:08.963135 12899 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:08.964892 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:08.965862 12742 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:08.974272 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:08.974977 12825 raft_consensus.cc:3055] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:08.975317 12825 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:08.951146 14065 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:08.951972 14410 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:09.004597 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 56375.2us
I20260504 20:01:09.005745 12716 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:08.997568 12825 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.011871 12825 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:09.012836 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:09.013563 12749 raft_consensus.cc:2393] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:08.941102 14438 consensus_queue.cc:1048] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260504 20:01:08.961931 14146 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260504 20:01:09.029948 12716 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.034701 12716 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.046114 12749 raft_consensus.cc:1275] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:09.050392 12743 raft_consensus.cc:1275] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:09.051239 12716 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.057006 12716 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.019033 14335 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:09.068395 12716 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.016390 14347 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
I20260504 20:01:09.081092 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:09.042378 14182 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.021476 14372 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:09.054464 14435 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:09.094064 14372 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:09.094483 12818 raft_consensus.cc:1275] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260504 20:01:09.098182 12818 pending_rounds.cc:85] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 1
I20260504 20:01:09.095158 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:09.064962 14355 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.103432 12749 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:09.098450 12818 pending_rounds.cc:107] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260504 20:01:09.110392 12818 raft_consensus.cc:2889] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:09.111218 14637 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:09.112038 14637 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.109490 12742 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:09.114100 14636 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:09.114920 14636 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:09.064249 14519 consensus_queue.cc:1059] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260504 20:01:09.061240 14376 consensus_queue.cc:1059] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260504 20:01:09.100176 14438 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:09.100464 14065 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.145504 12825 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:09.081672 14335 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.174737 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:09.179600 12749 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:09.181284 14335 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:09.182824 14335 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:09.183871 14335 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.119082 14569 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.137691 14347 consensus_queue.cc:1048] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260504 20:01:09.190141 14335 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:09.207652 12749 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
W20260504 20:01:09.227154 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 156617us
I20260504 20:01:09.226159 14410 raft_consensus.cc:697] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:09.227773 14335 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:09.228879 14636 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:09.319259 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:09.317523 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 279614us
W20260504 20:01:09.320088 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 251810us
I20260504 20:01:09.239944 14637 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:09.329195 12863 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:09.318780 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:09.335177 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:09.340132 13532 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260504 20:01:09.349433 14410 consensus_queue.cc:237] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:09.350437 14639 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
W20260504 20:01:09.351585 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:09.352612 14628 raft_consensus.cc:3055] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:09.353062 14628 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:09.355320 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:09.357388 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:09.359023 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:09.360458 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:09.362053 14636 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:09.365126 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:09.364989 14636 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.367825 14636 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:09.380210 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:09.377235 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:09.381469 12746 raft_consensus.cc:2393] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
W20260504 20:01:09.382097 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 52987.2us
W20260504 20:01:09.383033 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 50506.3us
I20260504 20:01:09.378727 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:09.323508 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.397647 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:09.380209 14634 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.258s user 0.001s sys 0.000s
I20260504 20:01:09.425796 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:09.434242 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:09.435461 12819 raft_consensus.cc:2393] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:09.439788 12817 raft_consensus.cc:3055] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:09.440392 12817 raft_consensus.cc:740] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:09.445396 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:09.327791 14335 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.357895 14629 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.072s
I20260504 20:01:09.358130 14637 consensus_queue.cc:1048] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
W20260504 20:01:09.454490 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 66976.5us
W20260504 20:01:09.447782 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 124435us
I20260504 20:01:09.411048 14641 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:09.454027 12817 consensus_queue.cc:260] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.464475 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:09.466995 13528 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:09.486667 12821 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:09.487529 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 47444.5us
I20260504 20:01:09.499423 12865 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:09.460197 14435 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:09.426440 14229 consensus_queue.cc:1059] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260504 20:01:09.522213 12865 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:09.536036 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 291508us
I20260504 20:01:09.446575 14656 consensus_queue.cc:1059] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.102s
I20260504 20:01:09.509219 14444 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:09.511392 14632 raft_consensus.cc:3055] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:09.588675 14632 raft_consensus.cc:740] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:09.588912 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:09.589593 13532 raft_consensus.cc:1275] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:09.535115 14636 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.448457 14628 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.535356 14302 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260504 20:01:09.535719 14519 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260504 20:01:09.509652 14376 consensus_queue.cc:1059] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260504 20:01:09.537046 14641 raft_consensus.cc:697] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:09.598402 14648 log.cc:927] Time spent T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260504 20:01:09.597851 14641 consensus_queue.cc:237] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.535898 14349 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260504 20:01:09.571280 14410 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:09.581677 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 46400.6us
I20260504 20:01:09.584321 13499 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:09.510330 14569 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
W20260504 20:01:09.622857 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 104954us
I20260504 20:01:09.539685 14438 consensus_queue.cc:1048] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260504 20:01:09.623644 12791 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:09.650851 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 128720us
I20260504 20:01:09.621376 14658 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:09.574072 14631 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260504 20:01:09.652937 12865 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:09.641093 14229 raft_consensus.cc:3055] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:09.654942 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:09.658665 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 52590.9us
I20260504 20:01:09.659404 12791 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:09.644380 14355 consensus_queue.cc:1059] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260504 20:01:09.656554 14229 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:09.661144 14229 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.662456 14229 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:09.663053 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:09.616782 14444 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:09.656457 14347 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.670781 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:09.671447 12819 raft_consensus.cc:606] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.643102 14636 consensus_queue.cc:1048] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.016s
I20260504 20:01:09.657987 14658 raft_consensus.cc:697] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:09.695079 12742 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260504 20:01:09.664850 14435 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:09.703060 14658 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:09.703305 14435 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:09.656635 14335 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:09.589672 14632 consensus_queue.cc:260] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.699286 12825 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:09.711253 14519 raft_consensus.cc:3055] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:09.711746 14632 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:09.671736 14656 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:09.711931 14519 raft_consensus.cc:740] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:09.710769 14647 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:09.712900 14647 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.718731 14647 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:09.711846 12825 raft_consensus.cc:606] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.720081 14628 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:09.697921 14633 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:09.722265 14410 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:09.723341 12896 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260504 20:01:09.727149 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 52664.8us
I20260504 20:01:09.729239 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:09.732836 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:09.741963 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 69538us
I20260504 20:01:09.742972 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:09.743209 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:09.743685 12819 raft_consensus.cc:606] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.744015 14444 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:09.745777 14435 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:09.713369 14519 consensus_queue.cc:260] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.747390 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:09.747714 14519 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:09.747766 12817 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:09.754875 14652 raft_consensus.cc:3055] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:09.755338 14652 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:01:09.758872 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 99570.4us
I20260504 20:01:09.762403 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:09.762166 12817 raft_consensus.cc:1275] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:09.768159 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:09.772754 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:09.769860 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:09.790865 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:09.791957 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:09.790776 12750 raft_consensus.cc:1240] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:09.800663 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:09.804582 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:09.830677 12816 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
W20260504 20:01:09.835268 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 52257.1us
I20260504 20:01:09.780375 14146 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:09.775863 14656 raft_consensus.cc:697] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:09.789350 14444 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.856053 12749 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:09.852752 12892 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:09.853217 12791 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:09.770938 14652 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:09.823448 14410 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:09.834867 14633 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:09.881515 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:09.848466 14653 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:09.890882 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:09.910993 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:09.869277 14675 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:09.913110 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:09.923775 12791 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:09.885005 14302 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:09.884011 14444 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:09.883317 14652 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:09.932783 12825 raft_consensus.cc:1275] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:09.890028 14065 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:09.912211 14631 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.878919 14410 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:09.947830 12749 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:09.919574 14629 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:09.929728 14675 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:09.951676 14444 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:09.956766 14065 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:10.004207 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 164077us
I20260504 20:01:09.940635 14645 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:10.030526 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 76734.5us
I20260504 20:01:09.955835 14604 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260504 20:01:10.033751 12822 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
W20260504 20:01:10.035458 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 78957.6us
I20260504 20:01:09.994910 14302 raft_consensus.cc:697] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:10.053644 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 165149us
I20260504 20:01:10.059798 12749 raft_consensus.cc:1275] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:10.062937 12791 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:09.902638 14633 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.065477 12716 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:09.953446 14660 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.300s
I20260504 20:01:10.085574 14644 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:10.086735 14683 consensus_queue.cc:1048] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260504 20:01:10.087201 14682 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:10.088416 14680 log.cc:927] Time spent T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
W20260504 20:01:10.091452 14678 log.cc:927] Time spent T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
W20260504 20:01:10.091773 14679 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.312s user 0.002s sys 0.000s
W20260504 20:01:10.092480 14674 log.cc:927] Time spent T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.229s user 0.001s sys 0.001s
I20260504 20:01:10.093140 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:10.093766 14675 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.055155 14645 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.096297 14645 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.096855 14645 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.099115 14633 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.080749 14676 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.105827 13528 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:10.097345 14645 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.129220 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.088436 14683 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:10.134886 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 72033.3us
I20260504 20:01:10.130873 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:10.135797 12791 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:10.032402 14656 consensus_queue.cc:237] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.136232 13528 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:10.141330 12864 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:10.080951 14372 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260504 20:01:10.149056 12893 raft_consensus.cc:1240] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 4. Ops: []
I20260504 20:01:10.165194 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.083056 14649 consensus_queue.cc:1048] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.269s
I20260504 20:01:10.173049 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:10.161293 14632 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:10.163883 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.179615 14641 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.189924 14146 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.190363 14684 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
I20260504 20:01:10.191567 14693 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:10.193847 14695 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:10.219733 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.220393 14693 raft_consensus.cc:697] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:10.214792 14632 raft_consensus.cc:697] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:10.220955 14065 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:10.169914 14302 consensus_queue.cc:237] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.201136 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.141746 14683 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:10.234364 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 90914.7us
I20260504 20:01:10.121073 14676 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:10.218464 14410 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:10.218540 14695 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:10.215081 12820 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:10.218837 14444 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.219524 14146 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.216228 14641 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.257462 12863 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:10.257833 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:10.262959 12894 raft_consensus.cc:1275] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:10.264266 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:10.278507 12897 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 2: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:10.282202 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:10.283246 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:10.258962 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:10.245045 14652 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:10.291323 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:10.294734 12823 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 6
I20260504 20:01:10.295128 12824 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260504 20:01:10.264657 14683 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:10.247519 12633 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 0e8eba4dbb8d436f81062ca3bbf46788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 5ff0ef72ec3b4667893a0420d3c7f923 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:10.299111 14683 consensus_queue.cc:1048] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260504 20:01:10.299556 12633 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 57076.7us
I20260504 20:01:10.301017 12747 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:01:10.301858 14676 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.302815 14676 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:10.303241 14676 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:10.303679 14676 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:10.305581 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:10.248462 14695 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:10.305922 14670 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:10.306376 13528 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:10.221650 14693 consensus_queue.cc:237] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.306567 14670 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:10.307494 14641 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.308148 14641 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:10.308780 14641 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:10.325783 12864 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:10.309275 14641 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:10.326236 14628 consensus_meta.cc:287] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.606s user 0.013s sys 0.001s
W20260504 20:01:10.332958 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 73958.3us
I20260504 20:01:10.332775 14645 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.347684 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:10.348766 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 91409.4us
I20260504 20:01:10.353027 12816 raft_consensus.cc:606] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:10.351787 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.353555 12816 raft_consensus.cc:614] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:10.354202 12818 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:10.373885 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:10.405354 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 51712us
W20260504 20:01:10.405377 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 48450.5us
I20260504 20:01:10.322775 14652 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.411732 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.344056 14410 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:10.341624 14146 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:10.413265 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.410931 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:10.423413 12747 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:10.452072 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:10.456717 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:10.355003 14670 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.452976 12897 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:10.353215 14376 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20260504 20:01:10.424014 14410 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:10.424423 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.334990 14632 consensus_queue.cc:237] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.471819 14713 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:10.471645 14705 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.129s
I20260504 20:01:10.477356 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:10.517428 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 51009.6us
I20260504 20:01:10.525178 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.461484 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:10.527768 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 168616us
W20260504 20:01:10.473737 14692 log.cc:927] Time spent T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.237s user 0.001s sys 0.000s
W20260504 20:01:10.474937 14688 log.cc:927] Time spent T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.538s user 0.005s sys 0.006s
I20260504 20:01:10.532377 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:10.534651 12716 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:10.535738 14146 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:10.536561 14723 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260504 20:01:10.476362 12820 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
W20260504 20:01:10.537906 14702 log.cc:927] Time spent T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.120s user 0.000s sys 0.001s
I20260504 20:01:10.538030 14716 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:10.475741 14689 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260504 20:01:10.540176 12823 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:01:10.540952 12716 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:10.474712 14698 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:10.526544 14693 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.469105 14410 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:10.538710 14716 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.475064 14714 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:10.536563 14146 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.554622 14721 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:10.521026 14652 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:10.558917 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:10.475389 14711 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:10.572165 12716 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:10.580442 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 174150us
I20260504 20:01:10.543744 14713 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.568708 14721 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:10.577845 14712 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260504 20:01:10.562713 14647 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:10.602434 12865 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:10.556596 14645 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:10.577903 14683 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:10.593216 14652 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:10.593674 14721 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:10.617033 14638 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.618353 12744 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:01:10.620057 14707 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:10.633486 14718 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260504 20:01:10.634292 14728 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.634989 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:10.637081 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:10.657269 12746 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:10.568804 14693 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:10.643007 14716 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.613966 14683 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:10.677148 14683 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:10.645323 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:10.637178 14637 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:10.621490 14641 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.562606 14698 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.660848 14728 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:10.667493 14676 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.669499 14146 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:10.638245 14638 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:10.570245 14714 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.661031 14707 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:10.672575 12744 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:10.614739 14645 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.688127 12819 raft_consensus.cc:606] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:10.702533 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.766189 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.749090 12823 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:01:10.769292 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:10.591615 14711 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.752696 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:10.753381 12863 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:10.754106 14693 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.754905 14728 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.771354 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:10.701629 14716 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:10.778056 12742 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:10.753392 14698 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.826781 14410 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:10.827347 14714 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.770444 14695 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:10.830518 12899 raft_consensus.cc:2368] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:10.833171 14695 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.772845 14631 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260504 20:01:10.834712 14638 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.849647 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:10.850441 13528 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:10.822000 14641 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:10.864574 14652 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.865430 14652 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.867739 14707 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.868311 14652 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:10.865276 14676 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.837435 14410 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:10.882256 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:10.869650 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:10.825009 14716 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.869047 14652 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:10.894933 13525 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:10.897065 14716 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.903882 14653 raft_consensus.cc:993] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:10.880591 14713 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.892026 14647 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:10.905315 14569 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:10.927177 14569 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.929540 14569 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:10.913569 14683 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.932329 14683 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.937631 14652 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:10.940821 14652 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:10.944008 14711 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:10.947880 13528 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:10.949584 12864 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:10.933362 14721 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.027266 14646 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260504 20:01:11.030440 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 80963.7us
I20260504 20:01:11.030750 14734 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:11.031776 14734 raft_consensus.cc:697] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:11.032574 14732 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.080s user 0.003s sys 0.000s
W20260504 20:01:11.036715 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 215416us
I20260504 20:01:11.040158 14335 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:11.123443 12865 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.014s sys 0.040s
W20260504 20:01:11.130803 12865 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.516s user 0.014s sys 0.040s
I20260504 20:01:11.121184 14444 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:11.049407 14376 consensus_queue.cc:1059] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20260504 20:01:11.146420 14704 raft_consensus.cc:1081] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:11.125777 14335 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.148598 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:11.150424 14684 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:11.036159 14734 consensus_queue.cc:237] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:11.151554 14355 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:11.158711 13525 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:11.125095 14647 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:11.172353 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 569970us
I20260504 20:01:11.211874 12865 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:11.187150 14702 log.cc:927] Time spent T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260504 20:01:11.188256 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1547) took 1039 ms. Trace:
I20260504 20:01:11.214888 12818 rpcz_store.cc:276] 0504 20:01:10.128751 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.128932 (+ 181us) service_pool.cc:224] Handling call
0504 20:01:11.168302 (+1039370us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54000a4bd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:11.168482 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":515367808}
I20260504 20:01:11.228914 12864 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:11.271754 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1613) took 1385 ms. Trace:
I20260504 20:01:11.272316 12892 rpcz_store.cc:276] 0504 20:01:09.847458 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:09.889154 (+ 41696us) service_pool.cc:224] Handling call
0504 20:01:09.905516 (+ 16362us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:10.149069 (+243553us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:10.149108 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:10.164571 (+ 15463us) log.cc:844] Serialized 26 byte log entry
0504 20:01:10.432292 (+267721us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:10.466577 (+ 34285us) log.cc:844] Serialized 14 byte log entry
0504 20:01:10.476921 (+ 10344us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:10.618692 (+141771us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b18003483a0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159910b62 00007f0159912e01 00007f0159912db9 00007f0159912d49 00007f0159912d11 00007f015991200d 0000000000379254 0000000000379099 00007f01598f3fba
0504 20:01:11.232148 (+613456us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54002f71d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:11.232228 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:11.232295 (+ 67us) raft_consensus.cc:1685] finished
0504 20:01:11.232316 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:11.232513 (+ 197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10049,"prepare.queue_time_us":36377,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":98174,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":8277,"spinlock_wait_cycles":375945344,"thread_start_us":314578,"threads_started":2,"wal-append.queue_time_us":359052}
W20260504 20:01:11.285358 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 56538.4us
I20260504 20:01:11.290181 14683 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:11.290967 14687 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:11.291335 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:11.292156 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:11.216352 14707 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:11.292949 12891 raft_consensus.cc:2393] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:11.295717 12750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:01:11.298237 12750 raft_consensus.cc:606] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:11.296725 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1531) took 1043 ms. Trace:
I20260504 20:01:11.298583 12750 raft_consensus.cc:614] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:11.299582 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:11.191751 14444 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:11.283622 14636 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.355s user 0.017s sys 0.027s
I20260504 20:01:11.306368 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:11.307314 12897 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:11.300556 12893 rpcz_store.cc:276] 0504 20:01:10.252943 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.253100 (+ 157us) service_pool.cc:224] Handling call
0504 20:01:10.290783 (+ 37683us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:10.310638 (+ 19855us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:10.310670 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:10.325945 (+ 15275us) log.cc:844] Serialized 26 byte log entry
0504 20:01:10.518350 (+192405us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:10.518616 (+ 266us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:11.286695 (+768079us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b54000f4454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:11.286792 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:11.286861 (+ 69us) raft_consensus.cc:1685] finished
0504 20:01:11.286884 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:11.296642 (+ 9758us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59544,"raft.queue_time_us":23583,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":12248,"spinlock_wait_cycles":44678528,"thread_start_us":164843,"threads_started":1,"wal-append.queue_time_us":301685}
I20260504 20:01:11.210042 14355 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.318771 14065 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:11.291824 14728 raft_consensus.cc:3055] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:11.220221 14647 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:11.312253 14636 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.388s user 0.017s sys 0.028s
W20260504 20:01:11.313699 12747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.040s sys 0.079s
W20260504 20:01:11.335896 12747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.041s sys 0.079s
I20260504 20:01:11.327950 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.336879 12818 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:01:11.336773 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 377724us
I20260504 20:01:11.296479 14719 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:11.319470 14065 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:11.300768 14714 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:11.263741 14641 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260504 20:01:11.210068 14684 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.342753 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260504 20:01:11.350684 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.348448 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.357309 12743 raft_consensus.cc:2393] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:11.291647 14687 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:11.320540 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:11.290984 14683 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:11.325781 14632 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:11.330837 14229 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:11.329708 14728 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:11.381773 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:11.389052 14065 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:11.434610 14744 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:11.432076 14650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.002s sys 0.024s
I20260504 20:01:11.343611 14348 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:11.351836 14741 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:11.372596 14724 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:11.454427 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.459868 14647 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:11.460621 14735 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260504 20:01:11.460516 14355 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:11.382189 14714 raft_consensus.cc:697] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:11.363560 14719 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:11.483659 12790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.014s sys 0.030s
I20260504 20:01:11.393925 14632 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:11.474983 14650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s user 0.003s sys 0.026s
I20260504 20:01:11.414779 14229 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:11.489004 12790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s user 0.014s sys 0.030s
I20260504 20:01:11.474866 12892 raft_consensus.cc:1240] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:11.523545 12863 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:11.530296 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:11.471531 14628 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:11.475272 14347 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:11.596594 12863 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:11.491317 14731 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:11.617017 12824 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:11.626034 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:11.471824 14348 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:11.468812 14744 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:11.504107 14335 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:11.503541 14745 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:11.659138 14750 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:11.660897 14736 log.cc:927] Time spent T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.065s user 0.002s sys 0.002s
I20260504 20:01:11.662184 14229 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:11.662258 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.683393 12823 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:11.663712 14687 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:11.663784 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.686502 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 212) took 1337 ms. Trace:
I20260504 20:01:11.686806 12822 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:11.679663 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.688536 12816 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:11.604156 14641 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:11.626565 14716 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:11.508749 14728 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.546563 14636 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:11.700235 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.687033 12896 rpcz_store.cc:276] 0504 20:01:10.348786 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.348985 (+ 199us) service_pool.cc:224] Handling call
0504 20:01:10.349329 (+ 344us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:10.528012 (+178683us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:10.528050 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:10.546824 (+ 18774us) log.cc:844] Serialized 26 byte log entry
0504 20:01:11.034216 (+487392us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:11.034263 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:11.034339 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:11.686166 (+651827us) raft_consensus.cc:1685] finished
0504 20:01:11.686195 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:11.686394 (+ 199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":124666,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":612635,"threads_started":2,"wal-append.queue_time_us":725121}
I20260504 20:01:11.621266 14647 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:11.701709 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 382701us
I20260504 20:01:11.701180 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.477651 14741 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:11.594787 14734 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20260504 20:01:11.487789 14724 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.620942 14628 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.725749 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:11.744683 12816 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 3
I20260504 20:01:11.701110 14335 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:11.703971 14444 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:11.620839 14347 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.703421 14716 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:11.749030 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:11.758780 12823 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 4
I20260504 20:01:11.708642 14229 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:11.765314 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:11.734036 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 365718us
I20260504 20:01:11.769754 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:11.770491 12749 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:11.773731 12818 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:11.681170 14731 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.836114 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:11.837683 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 64321us
I20260504 20:01:11.837119 12821 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:11.707121 14641 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:11.834547 12749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1408) took 1519 ms (client timeout 1700 ms). Trace:
I20260504 20:01:11.850147 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.764070 14716 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:11.768497 14683 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:11.692485 14745 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.876101 12823 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:11.723155 14647 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.770071 14444 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:11.895635 12816 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:11.758175 14636 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:11.850232 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:11.838245 14229 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.772305 14728 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:11.866729 12749 rpcz_store.cc:269] 0504 20:01:10.251517 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.251680 (+ 163us) service_pool.cc:224] Handling call
0504 20:01:11.769578 (+1517898us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b5800000f98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d39ec 00007f01598d35ab 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:11.770925 (+ 1347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9281152}
I20260504 20:01:11.891458 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:11.687167 14714 consensus_queue.cc:237] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.028589 14229 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:12.033450 12899 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1530) took 1780 ms (client timeout 1655 ms). Trace:
I20260504 20:01:12.034149 14747 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:12.034116 12899 rpcz_store.cc:269] 0504 20:01:10.252581 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.252899 (+ 318us) service_pool.cc:224] Handling call
0504 20:01:12.033043 (+1780144us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54002cd3d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:12.033236 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1235489536}
I20260504 20:01:12.035422 14747 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:12.039448 12743 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1405) took 1791 ms (client timeout 1528 ms). Trace:
W20260504 20:01:12.039914 12743 rpcz_store.cc:269] 0504 20:01:10.247657 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.257251 (+ 9594us) service_pool.cc:224] Handling call
0504 20:01:11.148340 (+891089us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b5800000f98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:12.039167 (+890827us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54000dcad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:12.039359 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381192320}
I20260504 20:01:11.762424 14335 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:11.848809 14684 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:11.774407 14672 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:11.838495 14704 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:11.712119 14750 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:11.868124 14649 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:11.973433 14719 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.021526 12820 raft_consensus.cc:2410] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260504 20:01:12.049764 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.026454 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.078176 12817 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:12.075749 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:12.081869 14746 log.cc:927] Time spent T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.544s user 0.001s sys 0.000s
I20260504 20:01:12.083887 13532 raft_consensus.cc:2368] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
W20260504 20:01:12.088099 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 414655us
I20260504 20:01:11.894690 14687 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.167s
W20260504 20:01:12.065573 14605 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.045s sys 0.055s
I20260504 20:01:12.098196 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.062341 14632 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.053519 14636 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.075510 14704 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.118203 12714 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:12.093152 14348 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.053301 14444 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.136327 12714 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:12.085491 14649 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:12.105027 14605 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.641s user 0.046s sys 0.056s
I20260504 20:01:12.138454 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 193) took 1596 ms. Trace:
W20260504 20:01:12.139550 12742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1606) took 1371 ms (client timeout 1744 ms). Trace:
W20260504 20:01:12.148840 12742 rpcz_store.cc:269] 0504 20:01:10.767829 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.769040 (+ 1211us) service_pool.cc:224] Handling call
0504 20:01:12.139258 (+1370218us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x7b5400242a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:12.139447 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18277248}
I20260504 20:01:12.117856 14760 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:12.149933 13499 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.047s sys 0.092s
W20260504 20:01:12.155966 13499 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s user 0.047s sys 0.092s
I20260504 20:01:12.089130 14672 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.168426 14759 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:12.169143 14672 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.144826 12748 rpcz_store.cc:276] 0504 20:01:10.542601 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.545154 (+ 2553us) service_pool.cc:224] Handling call
0504 20:01:10.545574 (+ 420us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:10.551309 (+ 5735us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:10.551368 (+ 59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:10.585991 (+ 34623us) log.cc:844] Serialized 52 byte log entry
0504 20:01:11.036442 (+450451us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:11.036495 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:11.036571 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:12.138110 (+1101539us) raft_consensus.cc:1685] finished
0504 20:01:12.138143 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:12.138365 (+ 222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5249,"raft.queue_time_us":8158,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":421031,"threads_started":1,"wal-append.queue_time_us":728273}
W20260504 20:01:12.157857 14756 log.cc:927] Time spent T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.282s user 0.000s sys 0.001s
W20260504 20:01:12.157258 14738 log.cc:927] Time spent T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.170s user 0.000s sys 0.001s
I20260504 20:01:12.142840 14628 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.186618 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.157053 14724 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.158358 14632 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:12.194077 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.157018 14647 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.154697 14744 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.181561 14659 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:12.200299 14750 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:12.200750 12715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.044s sys 0.066s
W20260504 20:01:12.211831 12715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.864s user 0.045s sys 0.066s
I20260504 20:01:12.200820 14704 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.203987 14649 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.204525 14645 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.169221 14759 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.206293 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:12.208772 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:12.171872 14741 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.230897 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.223707 12790 leader_election.cc:400] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:12.195462 14347 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.237016 13528 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260504 20:01:12.278136 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1602) took 1636 ms. Trace:
I20260504 20:01:12.205725 14641 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:12.296370 12716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.011s sys 0.045s
W20260504 20:01:12.304949 12716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s user 0.013s sys 0.045s
I20260504 20:01:12.210777 14632 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.278717 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:12.305399 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 585903us
W20260504 20:01:12.287727 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 84960.7us
I20260504 20:01:12.307603 14632 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.234685 14719 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:12.308358 12716 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:12.219632 14659 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.308771 14719 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.309513 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:12.310613 12899 raft_consensus.cc:2468] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
W20260504 20:01:12.287441 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 510699us
I20260504 20:01:12.307271 14641 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:12.205940 14747 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.311479 14641 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.311889 14719 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.217187 14744 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.269991 14745 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:12.287927 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 151664us
I20260504 20:01:12.313616 14641 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.217386 14647 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:12.235777 12790 leader_election.cc:403] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:12.232566 14731 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.304409 12747 rpcz_store.cc:276] 0504 20:01:10.641743 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.642602 (+ 859us) service_pool.cc:224] Handling call
0504 20:01:10.642980 (+ 378us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:10.643356 (+ 376us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:10.643386 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:10.644066 (+ 680us) log.cc:844] Serialized 26 byte log entry
0504 20:01:11.336342 (+692276us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:11.708951 (+372609us) log.cc:844] Serialized 14 byte log entry
0504 20:01:12.143968 (+435017us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:12.144052 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:12.277778 (+133726us) raft_consensus.cc:1685] finished
0504 20:01:12.277830 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:12.278033 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":638370,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":339920,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":4543,"thread_start_us":1486940,"threads_started":3,"wal-append.queue_time_us":843058}
I20260504 20:01:12.216118 14767 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:12.246996 14335 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:12.299081 12819 consensus_meta.cc:287] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.526s user 0.009s sys 0.002s
I20260504 20:01:12.317934 12716 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:12.165153 14760 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.320343 13499 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:12.314332 14347 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260504 20:01:12.230896 14724 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:12.315364 14659 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.327925 12714 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:12.331481 14647 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:12.336941 12819 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:12.322809 14741 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.325663 14636 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.327394 14637 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:12.328871 14672 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:12.335465 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 111947us
I20260504 20:01:12.346074 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.349207 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.382944 12790 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:12.349972 14727 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:12.350701 14760 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.342011 14761 raft_consensus.cc:2804] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260504 20:01:12.351833 14766 log.cc:927] Time spent T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.177s user 0.000s sys 0.001s
I20260504 20:01:12.354657 14759 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.334980 14444 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.339639 14728 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
I20260504 20:01:12.334560 14767 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:12.329288 14744 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.360119 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1411) took 1939 ms. Trace:
I20260504 20:01:12.361905 14647 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:12.338853 14731 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:12.366477 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.399106 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.364184 14774 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.400643 12745 rpcz_store.cc:276] 0504 20:01:10.420861 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.421014 (+ 153us) service_pool.cc:224] Handling call
0504 20:01:10.421426 (+ 412us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:11.894553 (+1473127us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b18003c98e0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f015c55585c 00007f01598f3c0b 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f015990da95 00007f015c9a6440 00007f015c9a6fdb 00007f015c9b1a2b
0504 20:01:11.972503 (+ 77950us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:11.972547 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:11.973198 (+ 651us) log.cc:844] Serialized 26 byte log entry
0504 20:01:12.283940 (+310742us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:12.284335 (+ 395us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:12.284604 (+ 269us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:12.359686 (+ 75082us) raft_consensus.cc:1685] finished
0504 20:01:12.359715 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:12.360031 (+ 316us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60,"raft.queue_time_us":35023,"raft.run_cpu_time_us":456,"raft.run_wall_time_us":55980,"spinlock_wait_cycles":5340416,"thread_start_us":414651,"threads_started":2,"wal-append.queue_time_us":346344}
I20260504 20:01:12.390515 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.391353 14761 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:12.390745 14632 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:12.384922 14727 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.395635 14760 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:12.391018 14605 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:12.348063 14724 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.443168 13530 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:12.446422 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.438994 14774 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.450004 14782 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.389123 14672 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.396824 14779 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:12.380416 14741 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:12.452306 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 124519us
I20260504 20:01:12.460589 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1541) took 1103 ms. Trace:
I20260504 20:01:12.385414 14637 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.452853 14719 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:12.454916 14761 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:12.471459 12893 rpcz_store.cc:276] 0504 20:01:11.357021 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:11.393744 (+ 36723us) service_pool.cc:224] Handling call
0504 20:01:11.462260 (+ 68516us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:11.462697 (+ 437us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:11.462730 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:11.473743 (+ 11013us) log.cc:844] Serialized 26 byte log entry
0504 20:01:11.729223 (+255480us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:11.729473 (+ 250us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:11.729575 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:12.460237 (+730662us) raft_consensus.cc:1685] finished
0504 20:01:12.460285 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:12.460497 (+ 212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":571,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":9796,"thread_start_us":177722,"threads_started":1,"wal-append.queue_time_us":625161}
I20260504 20:01:12.403596 14767 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:12.466305 14438 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:12.481796 14774 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.482288 14637 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.478534 14719 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:12.483770 14672 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.483614 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:12.485397 14737 log.cc:927] Time spent T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 1.053s user 0.003s sys 0.006s
I20260504 20:01:12.459438 14760 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:12.502707 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.471256 14780 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.459255 14632 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:12.497692 14716 consensus_meta.cc:287] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.592s user 0.015s sys 0.000s
I20260504 20:01:12.504550 14719 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:12.516448 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.521406 14783 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:12.522296 14770 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.522224 14783 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.508370 12745 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:12.501665 14146 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.470700 14782 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.513856 14716 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.515352 14744 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.527587 14778 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.529950 14724 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.506014 14785 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.523355 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.523010 14770 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.475324 14779 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.522516 12715 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:12.501935 14438 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.533116 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.553328 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.530575 14146 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.554225 12745 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:12.536541 14783 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.537135 14349 raft_consensus.cc:993] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:12.510789 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.544054 14744 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:12.570288 14744 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.573552 14376 raft_consensus.cc:1081] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:12.510383 14760 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:12.542425 14782 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.543745 14785 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.547978 14672 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:12.534909 13524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 315) took 2294 ms. Trace:
I20260504 20:01:12.556669 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.557566 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:01:12.560092 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.560195 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.565495 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:12.575731 13499 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:12.580654 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.590548 12746 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:12.553226 14438 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.589548 12743 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:12.589318 13524 rpcz_store.cc:276] 0504 20:01:10.240228 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:10.945511 (+705283us) service_pool.cc:224] Handling call
0504 20:01:10.946056 (+ 545us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:10.946380 (+ 324us) raft_consensus.cc:1534] Early marking committed up to index 7
0504 20:01:11.414548 (+468168us) log.cc:844] Serialized 14 byte log entry
0504 20:01:12.448055 (+1033507us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b1800223a80. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d5da 00007f015c9b8e0a
0504 20:01:12.510582 (+ 62527us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:12.513669 (+ 3087us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:12.513748 (+ 79us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:12.526560 (+ 12812us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:12.534797 (+ 8237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":211,"prepare.queue_time_us":413919,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":4642,"raft.queue_time_us":3916,"raft.run_wall_time_us":16107,"spinlock_wait_cycles":23280640,"thread_start_us":487624,"threads_started":2,"wal-append.queue_time_us":1113060}
I20260504 20:01:12.598461 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.600149 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:12.512881 14780 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.606158 12743 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:12.529749 14778 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.583416 14146 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.599659 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.623148 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.603250 14762 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:12.625677 12749 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:12.626185 12745 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:12.626528 14744 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.628576 12741 raft_consensus.cc:2393] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:12.590608 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:12.622915 14784 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:12.599686 14672 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.636045 14779 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.631426 14762 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:12.655346 14762 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:12.657910 14637 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:12.658535 14637 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.632826 14741 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.664884 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.665208 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.660614 14637 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.666702 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1629) took 1378 ms. Trace:
I20260504 20:01:12.668545 12821 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:12.668211 12898 rpcz_store.cc:276] 0504 20:01:11.288217 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:11.300172 (+ 11955us) service_pool.cc:224] Handling call
0504 20:01:11.321917 (+ 21745us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:11.336632 (+ 14715us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:11.336673 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:11.336715 (+ 42us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:11.336749 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:12.666393 (+1329644us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b54000f4e54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:12.666464 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:12.666629 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33274624}
I20260504 20:01:12.671270 14780 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.677182 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.678092 12825 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:12.678162 14716 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:12.678702 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.679450 12816 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:12.682276 14632 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.685137 14672 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:12.684717 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.677330 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.686946 12819 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:12.690614 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.655791 14761 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:12.696156 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.697861 14761 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:12.699544 12743 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:12.764359 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:12.778182 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 153827us
I20260504 20:01:12.832875 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:12.872962 14785 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:12.875514 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 94724.4us
I20260504 20:01:12.692193 14647 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:12.934350 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:12.938263 12899 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:12.938879 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.937372 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.960436 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.974902 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.976969 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.977293 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:12.985777 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.987653 12898 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37318
I20260504 20:01:12.988807 12898 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:01:12.989477 12898 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:12.948299 14731 consensus_meta.cc:287] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.520s user 0.012s sys 0.000s
I20260504 20:01:13.009440 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:12.993862 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.003845 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.005959 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.006223 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.007264 12892 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:01:13.012224 12897 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:13.016471 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:13.017668 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1627) took 1730 ms (client timeout 1772 ms). Trace:
I20260504 20:01:12.941155 14770 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.039625 12899 raft_consensus.cc:2468] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:13.021116 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.024405 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.026826 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.018241 13497 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.027001 13525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 490) took 1412 ms. Trace:
I20260504 20:01:13.028312 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.032236 14762 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.033263 14727 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.033861 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.034857 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.038889 12896 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:01:13.042825 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:13.018227 12895 rpcz_store.cc:269] 0504 20:01:11.287236 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:11.287384 (+ 148us) service_pool.cc:224] Handling call
0504 20:01:13.016244 (+1728860us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:13.017564 (+ 1320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7771904}
I20260504 20:01:13.045248 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.057370 12748 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:13.058246 12743 raft_consensus.cc:1240] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:13.053253 14633 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.060325 14633 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.061019 12895 raft_consensus.cc:1275] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:13.061440 14646 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:13.060339 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:12.982100 14716 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:13.063062 14633 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:13.038555 14731 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.073086 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.074807 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.075672 12822 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
W20260504 20:01:13.049381 12791 connection.cc:254] RPC call timeout handler was delayed by -1.261197669000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260504 20:01:13.049940 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 341325us
I20260504 20:01:12.878724 14719 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.053884 14752 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:13.051162 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:12.981977 14672 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.042237 13499 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:13.063259 14716 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.050029 12890 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:13.125221 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.126329 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:13.077024 14767 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.573s user 0.014s sys 0.000s
I20260504 20:01:13.045974 13525 rpcz_store.cc:276] 0504 20:01:11.614964 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:11.649250 (+ 34286us) service_pool.cc:224] Handling call
0504 20:01:11.660800 (+ 11550us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:12.099792 (+438992us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:12.110846 (+ 11054us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:12.116123 (+ 5277us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:12.123810 (+ 7687us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:13.026659 (+902849us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b54004b0354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:13.026724 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:13.026920 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":982205696}
I20260504 20:01:13.132773 12899 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37318
I20260504 20:01:13.133378 12899 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:01:13.133723 12899 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.134593 12791 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.135778 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.130559 12748 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260504 20:01:13.139735 12750 raft_consensus.cc:2368] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:13.139536 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.142159 14387 raft_consensus.cc:2804] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:13.143010 14387 raft_consensus.cc:697] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:13.143848 14716 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:13.123607 13524 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:13.147187 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.150956 14721 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.158532 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.159559 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:13.160705 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 118619us
I20260504 20:01:13.131042 14752 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.162268 12743 raft_consensus.cc:2393] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:13.163544 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.179312 12748 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:13.167162 14762 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:13.182777 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:13.184827 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 45848.5us
I20260504 20:01:13.188235 12899 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:13.191004 12750 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:13.114849 12898 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.191298 12714 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.170423 14778 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:13.220036 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.224822 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.138146 14672 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.083324 14760 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.227578 12890 raft_consensus.cc:1240] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:13.220034 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.132004 14767 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.229247 12742 raft_consensus.cc:2368] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5.
I20260504 20:01:13.229024 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:13.229233 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.234404 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.235291 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.236317 14632 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:13.144583 14387 consensus_queue.cc:237] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.238263 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.238758 14762 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:13.239435 14762 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.240823 12750 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:13.246112 14752 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:13.247747 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.237037 14632 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.253760 14632 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:13.254048 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 62838.3us
I20260504 20:01:13.254477 14721 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:13.255009 14721 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:13.255594 14721 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.256127 12750 raft_consensus.cc:2368] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:13.257043 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.258358 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.259024 12899 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:13.261521 12714 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.264707 12898 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:13.259104 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.272130 12714 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.275491 14731 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:13.275710 14672 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:13.276039 14731 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:13.276614 14731 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:13.279134 14744 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:13.279717 14744 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:13.279789 14647 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.280218 14744 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:13.281309 14767 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:13.228075 14760 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.287417 12714 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.268715 12897 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:01:13.287928 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.290553 13499 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:13.251487 14762 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.291590 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.274302 12865 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.258976 12750 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:13.297634 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.299646 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.305362 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.309406 12865 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.309657 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.310916 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:13.312399 12865 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:13.312563 12750 raft_consensus.cc:2376] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3. Re-sending same reply.
I20260504 20:01:13.308780 14682 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:13.317155 14682 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:13.317653 14682 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:13.318859 14775 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.862s user 0.009s sys 0.008s
I20260504 20:01:13.320622 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:13.321580 12749 raft_consensus.cc:2368] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:13.308651 14672 raft_consensus.cc:2764] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260504 20:01:13.321636 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.322459 14745 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:13.322664 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1612) took 1229 ms. Trace:
I20260504 20:01:13.322751 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.292534 14658 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:13.331441 12716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.986s user 0.018s sys 0.032s
I20260504 20:01:13.326251 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.328199 12865 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.323114 12817 rpcz_store.cc:276] 0504 20:01:12.093247 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.321285 (+1228038us) service_pool.cc:224] Handling call
0504 20:01:13.322581 (+ 1296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:13.336270 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.308974 14348 raft_consensus.cc:2804] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:13.323217 14633 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:13.339216 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.343453 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:13.347148 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.348918 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:13.326037 14672 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:13.341048 14658 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.350201 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:13.330600 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:13.327306 14605 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.312188 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.350850 14444 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.341745 14348 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:13.344785 14633 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.355465 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:13.358263 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.381346 12817 raft_consensus.cc:3055] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:13.381402 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.381856 12817 raft_consensus.cc:740] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:13.363215 12824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1604) took 1978 ms (client timeout 2510 ms). Trace:
W20260504 20:01:13.382943 12824 rpcz_store.cc:269] 0504 20:01:11.385359 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:11.386049 (+ 690us) service_pool.cc:224] Handling call
0504 20:01:13.362947 (+1976898us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540015d1d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:13.363095 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337998592}
I20260504 20:01:13.366891 12747 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:13.376492 12716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.031s user 0.019s sys 0.032s
W20260504 20:01:13.387537 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 1.04807e+06us
I20260504 20:01:13.232995 14751 consensus_queue.cc:1059] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20260504 20:01:13.389516 12817 consensus_queue.cc:260] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:13.395077 12817 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.378605 14348 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.380542 14633 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:13.400638 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.408608 12824 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:13.406127 14760 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.410774 14760 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.411345 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.413031 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.414927 14727 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.415650 14727 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.371095 14672 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.403759 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.425031 12749 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:13.436579 12715 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.439076 12714 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:13.413383 14760 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:13.442996 13499 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:13.423775 12820 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:13.451023 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.456732 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:13.457556 12749 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.467051 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:13.468017 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 331635us
I20260504 20:01:13.468937 12791 leader_election.cc:400] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:13.469287 12791 leader_election.cc:403] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:13.439249 14569 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:13.472941 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.452940 14647 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:13.448971 14727 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.473351 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:13.478039 12863 connection.cc:254] RPC call timeout handler was delayed by -1.0586472690000619s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
I20260504 20:01:13.478965 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.477522 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:13.481860 12791 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.486485 12863 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.487413 12817 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:13.489004 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.489886 12824 raft_consensus.cc:3055] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:13.483366 12714 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:13.491006 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 162912us
I20260504 20:01:13.490331 12824 raft_consensus.cc:740] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:13.492090 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.492343 12791 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.494316 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.473693 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:13.495563 13529 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 5
I20260504 20:01:13.492262 12824 consensus_queue.cc:260] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.488982 14716 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.509774 14632 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:13.509573 14716 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.475046 14644 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:13.511936 12865 leader_election.cc:400] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:13.493129 14376 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260504 20:01:13.512410 14745 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:13.530936 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:13.511297 12824 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.522724 12749 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:13.490716 14631 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:13.534804 14745 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.526592 12863 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.526794 14672 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:13.477290 14647 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:13.496634 14444 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:13.532645 12865 leader_election.cc:403] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260504 20:01:13.545145 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 50970.5us
I20260504 20:01:13.532392 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:13.546351 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.514264 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.466617 14721 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.543777 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.546056 14807 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:13.546532 14672 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:13.544471 14647 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:13.555313 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:13.555965 12891 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 1.
I20260504 20:01:13.474618 14799 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:13.537220 14752 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.492990 14229 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260504 20:01:13.551847 12749 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:13.554684 12822 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:13.552330 12817 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:13.552261 12865 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:13.540081 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 46019.2us
I20260504 20:01:13.562595 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.562611 12824 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:13.496629 14604 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260504 20:01:13.488500 14724 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:13.557227 13499 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:13.540946 14745 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.545234 14731 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.531807 14744 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.610683 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.554004 14807 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.555233 14729 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:13.562309 14799 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.586597 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.550104 14658 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.645535 12822 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:13.619709 12750 raft_consensus.cc:2368] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:13.622175 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:13.622397 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 65302.7us
I20260504 20:01:13.637156 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57934 (request call id 203) took 1121 ms. Trace:
I20260504 20:01:13.649564 12742 rpcz_store.cc:276] 0504 20:01:12.516052 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:12.516180 (+ 128us) service_pool.cc:224] Handling call
0504 20:01:13.636919 (+1120739us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b5400243954. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:13.637083 (+ 164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1637397760}
W20260504 20:01:13.611264 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 46479.9us
I20260504 20:01:13.651780 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:13.612468 14804 log.cc:927] Time spent T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20260504 20:01:13.634811 12820 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:13.616127 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 235) took 1069 ms. Trace:
I20260504 20:01:13.664197 12893 rpcz_store.cc:276] 0504 20:01:12.546791 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:12.546975 (+ 184us) service_pool.cc:224] Handling call
0504 20:01:12.575341 (+ 28366us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:12.683627 (+108286us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:12.683674 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:12.683722 (+ 48us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:12.683753 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:13.615863 (+932110us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b540013bfd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:13.615914 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:13.616066 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248506112}
I20260504 20:01:13.622509 14761 raft_consensus.cc:2804] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:13.618907 14731 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.644613 12747 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:13.662154 12865 leader_election.cc:400] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
W20260504 20:01:13.666126 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:13.667747 12824 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260504 20:01:13.679111 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:13.643779 12645 catalog_manager.cc:5671] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:13.680496 12865 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.658386 14644 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:13.687040 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.668300 14761 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:13.610844 14376 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.641523 14348 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.613447 14721 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.640404 13529 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:13.687743 12747 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:13.619403 14744 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.636040 14716 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:13.636029 14719 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.633770 14682 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.691740 12746 raft_consensus.cc:1240] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:13.717935 12818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1603) took 2341 ms (client timeout 1772 ms). Trace:
W20260504 20:01:13.749230 12818 rpcz_store.cc:269] 0504 20:01:11.350185 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:11.350353 (+ 168us) service_pool.cc:224] Handling call
0504 20:01:13.691815 (+2341462us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54003ae154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:13.691944 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":934865280}
I20260504 20:01:13.643476 14799 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:13.630187 14807 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:13.661144 14724 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:13.643154 14729 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:13.670964 14752 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:13.732525 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:13.751667 12822 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:13.751940 12817 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:13.686163 14644 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:13.720760 14444 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:13.768796 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 285501us
I20260504 20:01:13.775777 13499 leader_election.cc:400] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5.
I20260504 20:01:13.776327 13499 leader_election.cc:403] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:13.780189 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.780548 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.780898 12822 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:13.727586 14761 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.782173 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.785439 12818 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:13.759572 14729 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:13.793695 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 248632us
I20260504 20:01:13.756500 14724 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:13.766750 14644 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:13.774001 13530 raft_consensus.cc:1240] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:13.804287 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:13.801431 12863 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.812811 12714 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.813741 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:13.786105 14744 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.825248 12822 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:13.825822 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:13.824287 12818 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:13.800769 14649 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:13.767381 14752 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.784694 14765 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5.
I20260504 20:01:13.841684 12714 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.846984 12714 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.851959 14745 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.854988 14745 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:13.861135 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1419) took 1305 ms. Trace:
I20260504 20:01:13.861688 12745 rpcz_store.cc:276] 0504 20:01:12.555424 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:12.559839 (+ 4415us) service_pool.cc:224] Handling call
0504 20:01:13.860897 (+1301058us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b54001abb54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:13.861055 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83325312}
W20260504 20:01:13.862169 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 48614.5us
I20260504 20:01:13.863152 12863 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.858625 12714 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.855235 12791 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.841866 14721 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:13.873160 12791 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:13.876760 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 95401.1us
I20260504 20:01:13.832535 14229 consensus_queue.cc:1059] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260504 20:01:13.883298 12791 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.859174 14784 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:13.862504 14569 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:13.889124 14812 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:13.896508 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 216070us
I20260504 20:01:13.897358 12865 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:13.844957 14632 consensus_queue.cc:1059] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260504 20:01:13.889763 14745 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.877130 14525 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:13.890583 12863 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.855749 14376 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260504 20:01:13.875028 14631 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:13.903216 12791 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.880334 14444 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:13.906143 12863 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.907647 12865 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.913386 12863 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:13.943728 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 85219.5us
I20260504 20:01:13.903196 14569 raft_consensus.cc:697] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:01:13.924039 14814 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260504 20:01:13.944936 12714 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:13.912395 14525 raft_consensus.cc:697] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:13.937135 14682 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.906981 14811 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:13.950030 14823 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:13.950703 12865 leader_election.cc:400] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:13.951586 14649 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.951812 12863 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:13.898304 14721 raft_consensus.cc:697] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:13.922408 14820 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:13.963460 12865 leader_election.cc:403] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260504 20:01:13.910317 14716 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:13.969262 12745 raft_consensus.cc:3055] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:13.969401 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:13.979466 12745 raft_consensus.cc:740] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:13.978741 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.912580 14745 raft_consensus.cc:2764] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260504 20:01:13.952497 14784 raft_consensus.cc:697] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:13.908416 14348 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:13.978104 12863 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:13.973959 14633 consensus_meta.cc:287] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.538s user 0.010s sys 0.003s
I20260504 20:01:13.903550 14812 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:13.975231 14686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.007s user 0.008s sys 0.003s
I20260504 20:01:13.966100 14355 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:13.986361 12863 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:13.986708 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:13.986912 12745 consensus_queue.cc:260] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:13.987252 14525 consensus_queue.cc:237] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:13.987782 14633 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:14.001084 14686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.043s user 0.009s sys 0.003s
I20260504 20:01:13.989766 14752 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:14.010746 14784 consensus_queue.cc:237] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.012017 12745 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.988773 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:13.979674 14826 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:13.982148 14649 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:14.002229 14816 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:14.016036 14355 raft_consensus.cc:697] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:13.994309 14807 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.022931 14355 consensus_queue.cc:237] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.025177 14807 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:14.013737 12894 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:13.990387 14569 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:14.036362 14812 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:14.004099 14716 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260504 20:01:14.044323 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 58013.8us
W20260504 20:01:14.044932 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 58305.5us
I20260504 20:01:14.045398 12863 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:14.009356 14633 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:14.041041 12894 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:14.048266 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:14.062958 14807 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:14.063755 14807 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:14.076138 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:14.124230 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:14.154663 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 124461us
W20260504 20:01:14.155016 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 108820us
I20260504 20:01:14.156312 12863 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:14.157049 13499 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:14.064235 14807 raft_consensus.cc:697] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:14.048695 14716 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:14.160048 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 111895us
I20260504 20:01:14.009399 14729 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.172442 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:14.162756 12863 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:14.208565 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:14.209431 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 52492.9us
W20260504 20:01:14.172245 14387 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.016s sys 0.038s
W20260504 20:01:14.210260 14387 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s user 0.017s sys 0.038s
I20260504 20:01:14.211592 14822 raft_consensus.cc:2804] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:14.212316 14822 raft_consensus.cc:697] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
W20260504 20:01:14.214445 14830 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20260504 20:01:14.205446 14823 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:14.234762 14823 raft_consensus.cc:697] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:14.236255 14823 consensus_queue.cc:237] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.239571 14807 consensus_queue.cc:237] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.251338 14724 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:14.300238 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 137528us
I20260504 20:01:14.313764 14729 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:14.163743 14824 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:14.159336 14721 consensus_queue.cc:237] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:14.318423 14672 consensus_meta.cc:287] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.761s user 0.011s sys 0.001s
I20260504 20:01:14.319068 14672 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:14.159430 14633 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
W20260504 20:01:14.350106 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:14.168346 14716 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:14.374007 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:14.380378 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 110622us
I20260504 20:01:14.397384 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1440) took 1245 ms. Trace:
I20260504 20:01:14.207211 14745 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:14.397969 12743 rpcz_store.cc:276] 0504 20:01:13.152004 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.226318 (+ 74314us) service_pool.cc:224] Handling call
0504 20:01:14.373715 (+1147397us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b5800000f98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:14.397286 (+ 23571us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10423680}
I20260504 20:01:14.049232 14799 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:14.321089 14826 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 20:01:14.435055 14647 consensus_meta.cc:287] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.869s user 0.010s sys 0.003s
W20260504 20:01:14.446234 14831 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20260504 20:01:14.382536 14820 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:14.452833 14649 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:14.448736 14799 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:14.445281 14633 raft_consensus.cc:697] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:14.468118 14647 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:14.471174 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 298860us
I20260504 20:01:14.444832 14724 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:14.474069 13526 raft_consensus.cc:2393] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:14.213474 14822 consensus_queue.cc:237] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:14.489307 12865 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:14.495934 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:14.496145 14761 consensus_meta.cc:287] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.696s user 0.006s sys 0.006s
I20260504 20:01:14.483934 14633 consensus_queue.cc:237] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:14.506408 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:14.507454 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:14.511533 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:14.511734 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:14.509860 12747 raft_consensus.cc:1240] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:14.495046 14820 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:14.512737 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:14.516145 12744 raft_consensus.cc:1240] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:14.531342 12865 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.1:44007 timed out after 1.955s (SENT) [suppressed 9 similar messages]
I20260504 20:01:14.531607 12745 raft_consensus.cc:2368] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
I20260504 20:01:14.516695 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:14.515050 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:14.501461 14672 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:14.522921 14644 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.694s user 0.012s sys 0.000s
I20260504 20:01:14.502715 14649 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260504 20:01:14.525380 14444 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:14.534590 14820 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:14.527115 13524 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260504 20:01:14.511346 14816 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 20:01:14.541007 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:14.541509 14444 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.541971 14672 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.560472 14836 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:14.562170 14444 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:14.541606 14820 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:14.570830 14672 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:14.572101 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 56687.8us
I20260504 20:01:14.580108 14647 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:14.580919 12750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1456) took 1245 ms (client timeout 1591 ms). Trace:
W20260504 20:01:14.582250 14797 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 1.127s user 0.014s sys 0.014s
W20260504 20:01:14.583132 14835 log.cc:927] Time spent T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260504 20:01:14.584939 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 184) took 1882 ms. Trace:
I20260504 20:01:14.585908 13526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 501) took 1290 ms. Trace:
I20260504 20:01:14.586339 13526 rpcz_store.cc:276] 0504 20:01:13.295090 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.296160 (+ 1070us) service_pool.cc:224] Handling call
0504 20:01:13.530685 (+234525us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:14.585619 (+1054934us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b540057f8d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:14.585773 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2107115776}
I20260504 20:01:14.587899 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:14.588884 14837 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20260504 20:01:14.589105 13525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 505) took 1170 ms. Trace:
I20260504 20:01:14.590783 13532 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: []
W20260504 20:01:14.581511 12750 rpcz_store.cc:269] 0504 20:01:13.335351 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.335914 (+ 563us) service_pool.cc:224] Handling call
0504 20:01:14.580606 (+1244692us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400243454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:14.580817 (+ 211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":649611520}
I20260504 20:01:14.585369 12816 rpcz_store.cc:276] 0504 20:01:12.702408 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:12.702889 (+ 481us) service_pool.cc:224] Handling call
0504 20:01:12.703340 (+ 451us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:12.712946 (+ 9606us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:12.712991 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:12.957778 (+244787us) log.cc:844] Serialized 26 byte log entry
0504 20:01:13.327103 (+369325us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:13.327144 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:13.328434 (+ 1290us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:14.581555 (+1253121us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:14.584597 (+ 3042us) raft_consensus.cc:1685] finished
0504 20:01:14.584632 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:14.584849 (+ 217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16958,"raft.queue_time_us":9839,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146,"spinlock_wait_cycles":9828096,"thread_start_us":358191,"threads_started":1,"wal-append.queue_time_us":433387}
I20260504 20:01:14.589640 13525 rpcz_store.cc:276] 0504 20:01:13.419000 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.419213 (+ 213us) service_pool.cc:224] Handling call
0504 20:01:14.587707 (+1168494us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:14.589018 (+ 1311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20462336}
I20260504 20:01:14.596068 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:14.596661 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1578) took 1311 ms. Trace:
I20260504 20:01:14.512902 14761 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:14.597198 12819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1586) took 1294 ms (client timeout 1605 ms). Trace:
I20260504 20:01:14.601267 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.602944 12899 raft_consensus.cc:1240] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:14.598357 14816 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:14.606834 12896 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260504 20:01:14.600268 12819 rpcz_store.cc:269] 0504 20:01:13.302913 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.310696 (+ 7783us) service_pool.cc:224] Handling call
0504 20:01:14.595825 (+1285129us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:14.597111 (+ 1286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3274624}
I20260504 20:01:14.604708 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.597069 12821 rpcz_store.cc:276] 0504 20:01:13.284903 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.294904 (+ 10001us) service_pool.cc:224] Handling call
0504 20:01:13.363136 (+ 68232us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:13.363760 (+ 624us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:13.363827 (+ 67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:13.364513 (+ 686us) log.cc:844] Serialized 26 byte log entry
0504 20:01:13.492649 (+128136us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:13.492698 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:13.492771 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:13.623124 (+130353us) raft_consensus.cc:1685] finished
0504 20:01:13.623159 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:14.590123 (+966964us) spinlock_profiling.cc:243] Waited 67.9 ms on lock 0x7b540015d1d0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:14.596582 (+ 6459us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":493,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":174,"spinlock_wait_cycles":156175104,"thread_start_us":127606,"threads_started":1,"wal-append.queue_time_us":178155}
I20260504 20:01:14.610162 12898 raft_consensus.cc:1240] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:14.542519 14644 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.621667 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.622673 12898 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:14.625982 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:14.620903 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.629163 12896 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:14.640378 14761 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:14.644164 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:14.645053 12741 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:14.645653 12741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1429) took 1944 ms (client timeout 1955 ms). Trace:
W20260504 20:01:14.646131 12741 rpcz_store.cc:269] 0504 20:01:12.701095 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:12.705364 (+ 4269us) service_pool.cc:224] Handling call
0504 20:01:14.625429 (+1920065us) spinlock_profiling.cc:243] Waited 9.72 ms on lock 0x7b5800000f98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:14.645580 (+ 20151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22354304}
I20260504 20:01:14.647979 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.648514 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.648922 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.624123 14719 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:14.649300 12890 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:14.649873 14719 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.650812 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.649322 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.655912 12899 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:14.657310 12896 raft_consensus.cc:2393] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:14.658560 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.666991 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.675981 12891 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:14.679440 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:14.680671 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.682337 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.682547 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:14.683346 12899 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:14.678635 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.686388 12896 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:14.687498 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.688969 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.695982 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.616741 14816 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:14.703559 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:14.704542 12893 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:14.721544 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.727166 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:14.729974 14719 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:14.707042 14644 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:14.699704 14349 raft_consensus.cc:3055] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:14.731483 14349 raft_consensus.cc:740] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:14.732635 14349 consensus_queue.cc:260] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.737403 14349 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:14.748647 12715 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:14.755595 14768 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:14.759645 14834 log.cc:927] Time spent T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.264s user 0.000s sys 0.001s
I20260504 20:01:14.764492 12893 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:14.764758 13532 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 328) took 1436 ms. Trace:
I20260504 20:01:14.773590 13532 rpcz_store.cc:276] 0504 20:01:13.328251 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.328386 (+ 135us) service_pool.cc:224] Handling call
0504 20:01:14.557444 (+1229058us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:14.590472 (+ 33028us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:14.597715 (+ 7243us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:14.764521 (+166806us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b540057f8d0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:14.764681 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82367104}
I20260504 20:01:14.765558 14768 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:14.775967 12715 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:14.778416 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.788180 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.785842 13499 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:14.790827 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.792943 12899 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:14.794127 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:14.794770 12899 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:14.800930 12899 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:14.802440 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.803059 12898 raft_consensus.cc:2376] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3. Re-sending same reply.
I20260504 20:01:14.803916 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.804409 12898 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:14.809895 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:14.817199 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:14.769604 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1643) took 1336 ms. Trace:
I20260504 20:01:14.827494 12899 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:14.786605 14719 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:14.830773 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:14.836575 13532 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260504 20:01:14.837330 14833 log.cc:927] Time spent T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.352s user 0.001s sys 0.001s
I20260504 20:01:14.838791 14726 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:14.841665 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:14.826153 14660 raft_consensus.cc:2804] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:14.829101 12820 rpcz_store.cc:276] 0504 20:01:13.432846 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.720895 (+288049us) service_pool.cc:224] Handling call
0504 20:01:13.754084 (+ 33189us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:13.877449 (+123365us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:13.883219 (+ 5770us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:13.893351 (+ 10132us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:13.896970 (+ 3619us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:14.769195 (+872225us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54001973d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:14.769276 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:14.769495 (+ 219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":254948608}
I20260504 20:01:14.853595 12891 raft_consensus.cc:1240] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:14.856340 12899 raft_consensus.cc:1240] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:14.866017 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1626) took 1708 ms. Trace:
I20260504 20:01:14.866534 12748 rpcz_store.cc:276] 0504 20:01:13.157214 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.235598 (+ 78384us) service_pool.cc:224] Handling call
0504 20:01:13.862789 (+627191us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:13.862992 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:13.863016 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:13.863056 (+ 40us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:13.863088 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:13.863143 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:14.865714 (+1002571us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b54001abb50. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:14.865932 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1441586432}
I20260504 20:01:14.881119 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:14.828750 14820 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:14.883534 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:15.215643 12894 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:15.215961 12891 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:14.926559 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:15.216892 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 1.31369e+06us
I20260504 20:01:15.164158 12823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1575) took 2972 ms. Trace:
W20260504 20:01:14.970464 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 70109.5us
I20260504 20:01:14.924129 12748 raft_consensus.cc:2393] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:15.146593 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 194) took 1352 ms. Trace:
W20260504 20:01:15.115258 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:15.220086 12817 rpcz_store.cc:276] 0504 20:01:13.793779 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.832961 (+ 39182us) service_pool.cc:224] Handling call
0504 20:01:13.833305 (+ 344us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:13.833506 (+ 201us) raft_consensus.cc:1534] Early marking committed up to index 7
0504 20:01:13.856771 (+ 23265us) log.cc:844] Serialized 14 byte log entry
0504 20:01:13.857271 (+ 500us) log.cc:844] Serialized 14 byte log entry
0504 20:01:14.650238 (+792967us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b1800361f60. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159884e1a 00007f0162f265c9 00007f015c564110 00007f015c557633 00007f015c54d08a 00007f015c54d5da 00007f015c9b8e0a 00007f015c9c849f 00007f015c9c8418
0504 20:01:14.977704 (+327466us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b1800361f60. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159884e1a 00007f0162f265c9 00007f01608903ef 00007f0159910bb1 00007f0159912e01 00007f0159912db9 00007f0159912d49 00007f0159912d11 00007f015991200d
0504 20:01:15.146059 (+168355us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:15.146139 (+ 80us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:15.146210 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:15.146309 (+ 99us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:15.146491 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":547139,"prepare.queue_time_us":68523,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":96,"raft.queue_time_us":501012,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":10320,"spinlock_wait_cycles":41619584}
W20260504 20:01:15.051390 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 1.12422e+06us
W20260504 20:01:14.939106 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:14.981711 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:15.224658 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:15.148945 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:14.959540 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 75565.2us
I20260504 20:01:15.028406 12648 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36606 (request call id 118) took 6381 ms. Trace:
I20260504 20:01:15.229152 12743 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:15.235968 14719 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:15.236541 14719 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:15.126332 14843 log.cc:927] Time spent T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.446s user 0.004s sys 0.000s
W20260504 20:01:14.990319 14846 log.cc:927] Time spent T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.309s user 0.003s sys 0.000s
I20260504 20:01:14.865101 14660 raft_consensus.cc:697] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
W20260504 20:01:15.217291 14716 consensus_meta.cc:287] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.771s user 0.013s sys 0.001s
W20260504 20:01:15.004976 14842 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.325s user 0.002s sys 0.000s
I20260504 20:01:15.218463 12823 rpcz_store.cc:276] 0504 20:01:12.177161 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:12.456046 (+278885us) service_pool.cc:224] Handling call
0504 20:01:13.692070 (+1236024us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:13.793606 (+101536us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:13.793636 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:13.794163 (+ 527us) log.cc:844] Serialized 26 byte log entry
0504 20:01:13.907417 (+113254us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:13.923085 (+ 15668us) log.cc:844] Serialized 14 byte log entry
0504 20:01:15.048600 (+1125515us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:15.048727 (+ 127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:15.048775 (+ 48us) raft_consensus.cc:1685] finished
0504 20:01:15.048787 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:15.148854 (+100067us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54003ae150. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:15.149032 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1174342,"prepare.queue_time_us":3619,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":1155195,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":21939,"spinlock_wait_cycles":2841677056,"thread_start_us":165251,"threads_started":2,"wal-append.queue_time_us":117928,"wal-append.run_cpu_time_us":2146,"wal-append.run_wall_time_us":1005197}
I20260504 20:01:15.030226 14229 raft_consensus.cc:3055] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:14.957954 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 502) took 1655 ms. Trace:
I20260504 20:01:15.245306 12896 raft_consensus.cc:2393] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:15.233498 14726 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:14.987867 13525 raft_consensus.cc:1240] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:15.050714 14675 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:15.048113 14751 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:15.084522 14605 consensus_queue.cc:1059] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.170s
I20260504 20:01:15.225267 14816 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.247169 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:15.238309 14727 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:15.278177 14727 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.280799 14727 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:15.281448 12741 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:15.282482 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:15.283354 13525 raft_consensus.cc:2393] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:15.228550 12648 rpcz_store.cc:276] 0504 20:01:08.646653 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:08.646817 (+ 164us) service_pool.cc:224] Handling call
0504 20:01:08.654138 (+ 7321us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b540030f054. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c918470 00007f015c9b731a 00007f016275a46c 00007f01628543ee 00007f0160d3a4a4 00007f0160d3a432 00007f0160d3a361 00007f0160d3a2dc 00007f0160d39592 00007f015ace8bcc 00007f015ace8006
0504 20:01:15.028263 (+6374125us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:08.907461 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:08.908311 (+ 850us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:08.908363 (+ 52us) write_op.cc:435] Acquired schema lock
0504 20:01:08.908391 (+ 28us) tablet.cc:662] Decoding operations
0504 20:01:08.909333 (+ 942us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:08.909412 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:08.909455 (+ 43us) tablet.cc:685] Acquiring locks for 5 operations
0504 20:01:09.008508 (+ 99053us) tablet.cc:701] Row locks acquired
0504 20:01:09.012832 (+ 4324us) write_op.cc:260] PREPARE: finished
0504 20:01:09.026547 (+ 13715us) write_op.cc:270] Start()
0504 20:01:09.046592 (+ 20045us) write_op.cc:276] Timestamp: P: 1777924869022909 usec, L: 0
0504 20:01:09.051472 (+ 4880us) op_driver.cc:348] REPLICATION: starting
0504 20:01:09.145951 (+ 94479us) log.cc:844] Serialized 2039 byte log entry
0504 20:01:09.698479 (+552528us) op_driver.cc:464] REPLICATION: finished
0504 20:01:09.862736 (+164257us) write_op.cc:301] APPLY: starting
0504 20:01:09.872319 (+ 9583us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:09.903141 (+ 30822us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:09.953296 (+ 50155us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:10.102795 (+149499us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:10.103045 (+ 250us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0504 20:01:10.103097 (+ 52us) write_op.cc:312] APPLY: finished
0504 20:01:10.103376 (+ 279us) log.cc:844] Serialized 47 byte log entry
0504 20:01:10.103599 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:10.103759 (+ 160us) write_op.cc:454] Released schema lock
0504 20:01:10.119315 (+ 15556us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:10.160920 (+ 41605us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:10.165751 (+ 4831us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20290,"reported_tablets":13,"spinlock_wait_cycles":10852096,"child_traces":[["op",{"apply.queue_time_us":163962,"apply.run_cpu_time_us":4403,"apply.run_wall_time_us":274077,"num_ops":5,"prepare.queue_time_us":12343,"prepare.run_cpu_time_us":5925,"prepare.run_wall_time_us":299569,"replication_time_us":638008,"thread_start_us":90372,"threads_started":1}]]}
I20260504 20:01:15.277693 14820 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:15.239050 14640 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:15.291533 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:15.292177 12819 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:15.260725 12891 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:15.258659 14716 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.251405 12863 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:15.294739 12821 raft_consensus.cc:1275] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:15.245491 14811 consensus_queue.cc:1059] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260504 20:01:15.258605 13530 rpcz_store.cc:276] 0504 20:01:13.298903 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.304565 (+ 5662us) service_pool.cc:224] Handling call
0504 20:01:13.538265 (+233700us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:13.538530 (+ 265us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:13.830863 (+292333us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:14.884540 (+1053677us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54004b0fd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:14.953709 (+ 69169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":511061504}
I20260504 20:01:15.269302 14726 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.297515 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.298413 12825 raft_consensus.cc:2410] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:15.298743 14726 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:15.290800 14544 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:15.285123 14660 consensus_queue.cc:237] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.299707 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:15.258433 14229 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:15.276783 14818 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:15.300761 14818 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:15.301009 14229 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.301584 14605 raft_consensus.cc:3055] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:15.302094 14605 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:15.302167 14229 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:15.303131 14818 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:15.276892 12820 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:15.306699 14675 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:15.307265 14675 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:15.275043 12816 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:15.265446 14719 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.299232 14632 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:15.299260 14727 consensus_queue.cc:1059] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:15.294349 14815 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:15.278501 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:15.314633 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.307983 14675 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:15.312821 14747 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:15.316025 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:15.316408 14716 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:15.316315 14747 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.317306 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.317652 12825 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260504 20:01:15.318490 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:15.318596 14747 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:15.319284 12824 raft_consensus.cc:2368] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:15.319605 12893 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:15.320112 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:15.320797 12824 raft_consensus.cc:2368] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
I20260504 20:01:15.311069 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.292743 12823 raft_consensus.cc:1275] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:15.273226 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:15.312049 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.314756 14727 raft_consensus.cc:3055] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:15.318715 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:15.312141 12821 raft_consensus.cc:1275] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:15.325613 12817 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:15.338301 12893 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:15.339203 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.342381 12823 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260504 20:01:15.343452 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.341289 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.354238 12819 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:15.355785 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:15.356307 12633 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 0e8eba4dbb8d436f81062ca3bbf46788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 5ff0ef72ec3b4667893a0420d3c7f923 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:15.358541 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.358685 14818 consensus_queue.cc:1059] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260504 20:01:15.325459 14815 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.359877 14818 raft_consensus.cc:3055] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:15.360267 14818 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:15.327768 14605 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.329260 12820 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:15.308985 14640 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.299867 14544 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.362422 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:15.362828 12823 raft_consensus.cc:1275] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260504 20:01:15.363143 12817 raft_consensus.cc:2368] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:15.356460 12816 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:15.363451 14640 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:15.331700 14653 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:15.326028 14727 raft_consensus.cc:740] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:15.363157 12823 pending_rounds.cc:85] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 1
I20260504 20:01:15.364436 12823 pending_rounds.cc:107] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260504 20:01:15.364722 12823 raft_consensus.cc:2889] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:15.364256 14653 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.364823 14727 consensus_queue.cc:260] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.365926 13529 raft_consensus.cc:1240] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:15.366072 14727 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:15.367072 14653 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.322276 14632 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:15.359064 14832 log.cc:927] Time spent T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.093s user 0.002s sys 0.000s
I20260504 20:01:15.361140 14818 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.371889 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:15.372706 12748 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:15.374457 14818 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:15.376643 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:15.377215 14632 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.363451 14544 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:15.378222 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.380260 14675 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.382057 12822 raft_consensus.cc:1240] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:15.383360 14638 consensus_queue.cc:1048] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:01:15.386415 14768 consensus_queue.cc:1059] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:15.388214 14726 raft_consensus.cc:3055] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:15.387485 14768 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:15.390336 14605 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:15.391511 14719 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:15.395423 14653 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:15.395922 14631 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:15.395196 13527 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:15.396369 14632 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:15.400097 12741 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260504 20:01:15.401913 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:15.331866 14724 raft_consensus.cc:3055] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:15.405094 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:15.400942 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 57837.4us
I20260504 20:01:15.406492 12819 raft_consensus.cc:1275] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:15.437867 12742 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:15.409083 12863 leader_election.cc:400] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:15.438779 12863 leader_election.cc:403] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260504 20:01:15.408111 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:15.413059 12823 raft_consensus.cc:1240] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:15.408887 14726 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:15.361436 14815 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:15.450523 12748 raft_consensus.cc:2393] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 2: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:15.440055 12863 leader_election.cc:400] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
W20260504 20:01:15.451174 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 214438us
I20260504 20:01:15.453662 12863 leader_election.cc:403] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260504 20:01:15.444617 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 59165.2us
I20260504 20:01:15.438519 13497 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:15.457360 13528 raft_consensus.cc:1240] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:15.421324 14349 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:15.462266 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:15.443392 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:15.459154 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 227257us
I20260504 20:01:15.462975 12748 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:15.461508 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:15.463788 12865 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:15.465423 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:15.485863 12716 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:15.437301 14724 raft_consensus.cc:740] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:15.489624 13528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 503) took 2180 ms. Trace:
I20260504 20:01:15.451776 14672 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3.
I20260504 20:01:15.501978 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:15.504149 14863 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:15.505429 14660 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:01:15.506217 14863 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:15.496285 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:15.498971 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.457870 14807 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
I20260504 20:01:15.500949 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.433907 14653 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.436779 14631 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.437507 14376 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:15.508864 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 70443us
I20260504 20:01:15.514711 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:15.490316 13528 rpcz_store.cc:276] 0504 20:01:13.309504 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:13.310119 (+ 615us) service_pool.cc:224] Handling call
0504 20:01:14.979907 (+1669788us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:14.980179 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:15.457939 (+477760us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:15.489229 (+ 31290us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b54005c0154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:15.489522 (+ 293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1106891904}
I20260504 20:01:15.527190 14675 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:15.492794 14747 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:15.460839 14811 consensus_queue.cc:1059] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260504 20:01:15.439611 14632 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.458271 14544 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260504 20:01:15.491405 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:15.523114 14751 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:15.522022 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:15.547619 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:15.471551 14349 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.537235 14672 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:15.559253 14435 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:15.555137 14751 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.580461 14349 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.581132 14751 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:15.549235 14724 consensus_queue.cc:260] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.583039 14724 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:15.538192 14376 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:15.605433 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 141780us
I20260504 20:01:15.577685 14716 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:15.610617 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:15.594587 14229 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:15.613049 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 138857us
I20260504 20:01:15.601629 14435 consensus_queue.cc:1059] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260504 20:01:15.616065 14819 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:15.614939 14716 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:15.641300 12865 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:15.633713 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 166060us
I20260504 20:01:15.626783 14675 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:15.636531 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 181167us
I20260504 20:01:15.638800 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:15.652356 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 116878us
I20260504 20:01:15.656229 12716 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:15.659162 12714 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:15.658080 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:15.644507 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 126777us
I20260504 20:01:15.632758 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:15.662966 14864 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:15.632385 14653 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:15.650838 12863 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:15.645303 14672 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:15.646323 14865 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:15.651113 14435 consensus_queue.cc:1243] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer ef6057d9a49e4942b47ea0fcd66ab6f7 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260504 20:01:15.670727 12863 leader_election.cc:400] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:15.651970 14631 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.645561 14229 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.687294 14726 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:15.687995 14857 log.cc:927] Time spent T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.139s user 0.001s sys 0.003s
I20260504 20:01:15.688526 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:15.689107 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:15.648774 14820 consensus_queue.cc:1059] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20260504 20:01:15.689782 14875 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:15.669871 14632 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.694975 14716 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:15.696141 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:15.699952 14376 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.701730 14867 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:15.706642 12823 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:01:15.708834 14672 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:15.712845 12863 leader_election.cc:403] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260504 20:01:15.651253 14675 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.681197 12791 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:15.985654 14875 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:15.986302 14875 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:15.709578 14876 consensus_queue.cc:1048] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.281s
I20260504 20:01:15.804270 12749 raft_consensus.cc:1275] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:15.739589 14866 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:15.683828 14768 raft_consensus.cc:3055] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:15.993477 14768 raft_consensus.cc:740] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:15.994017 14820 raft_consensus.cc:3055] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:15.994454 14768 consensus_queue.cc:260] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:15.995613 14820 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:15.995975 12891 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:15.996177 14768 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:15.996564 14820 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:15.998221 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 340249us
I20260504 20:01:15.839689 14815 raft_consensus.cc:3055] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:15.999030 14815 raft_consensus.cc:740] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:15.999369 14675 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:15.999929 14815 consensus_queue.cc:260] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:15.820497 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 161452us
W20260504 20:01:15.751557 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:15.970657 14719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57450 (request call id 579) took 7135 ms. Trace:
I20260504 20:01:15.715380 14435 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:15.999195 12716 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:16.026870 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:16.008677 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:15.788477 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 140850us
I20260504 20:01:16.058292 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:16.058655 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 59557.3us
W20260504 20:01:16.057087 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 387816us
I20260504 20:01:16.061288 12716 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:15.822293 12818 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:15.770023 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:16.062017 12863 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:16.036679 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 355548us
I20260504 20:01:16.075654 12716 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:16.115017 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:15.842020 14229 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:16.001953 14815 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:16.022038 14724 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:16.187117 14724 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.037240 14644 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:16.121137 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:16.193975 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 72927.5us
I20260504 20:01:16.138304 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.078409 14716 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:16.047171 14653 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:16.127305 12865 leader_election.cc:400] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
I20260504 20:01:16.010895 14820 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:16.138862 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.186131 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.041236 14866 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:16.028116 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:16.184805 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.184442 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.189744 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:16.118868 13499 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:16.232643 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 113864us
I20260504 20:01:16.087419 14876 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:16.203402 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 76307.9us
I20260504 20:01:16.203899 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.065071 14882 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:16.046514 14605 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:16.204504 14750 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:16.059471 14672 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:16.238934 12865 leader_election.cc:400] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:16.051478 14875 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.121451 14819 consensus_queue.cc:1048] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20260504 20:01:16.220856 14865 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:16.244004 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:16.087982 14879 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:16.193212 14644 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:16.240413 14792 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:16.273721 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:16.274966 12825 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:16.275221 12791 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:16.183871 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.191339 14874 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:16.284221 12746 raft_consensus.cc:1240] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:16.265059 14672 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:16.264804 14823 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:16.181351 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:16.257339 12821 raft_consensus.cc:2955] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:16.276336 14792 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:16.354085 14792 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:16.354417 14860 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:16.355017 14860 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.357158 14860 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:16.177122 14816 consensus_queue.cc:1059] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260504 20:01:16.102681 14435 raft_consensus.cc:697] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:16.195492 14864 consensus_queue.cc:1048] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260504 20:01:16.191165 14807 consensus_queue.cc:1059] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260504 20:01:16.241820 14724 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:16.271831 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:16.115748 14880 raft_consensus.cc:2804] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:16.248468 14876 raft_consensus.cc:697] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:16.262472 14882 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:16.257103 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 138927us
I20260504 20:01:16.276587 14865 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.243292 14761 consensus_queue.cc:1048] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260504 20:01:16.277781 14644 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:16.236232 14866 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.274734 14811 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:16.238759 14653 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:16.258417 14605 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.350871 14867 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:16.351052 14875 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:16.369371 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:16.376153 14880 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:16.345952 14823 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.381613 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:16.382467 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:16.382886 12818 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:16.333099 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 218158us
I20260504 20:01:16.422859 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:16.423331 12716 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:16.423595 12819 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:16.392484 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 120794us
I20260504 20:01:16.366240 14752 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:16.365473 14376 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:16.056895 14719 rpcz_store.cc:276] 0504 20:01:08.835516 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:08.836400 (+ 884us) service_pool.cc:224] Handling call
0504 20:01:08.943397 (+106997us) log.cc:844] Serialized 487 byte log entry
0504 20:01:15.970482 (+7027085us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29676,"raft.queue_time_us":34850,"raft.run_cpu_time_us":13470,"raft.run_wall_time_us":289472,"thread_start_us":293104,"threads_started":2,"wal-append.queue_time_us":248524,"wal-append.run_cpu_time_us":2766,"wal-append.run_wall_time_us":1157183}
I20260504 20:01:16.426083 12825 raft_consensus.cc:606] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:16.426522 12825 raft_consensus.cc:614] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260504 20:01:16.406617 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 165305us
I20260504 20:01:16.378960 12714 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:16.428987 12865 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:16.430958 12716 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:16.432825 12716 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:16.435240 12865 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:16.434072 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 55247.1us
W20260504 20:01:16.426957 14885 log.cc:927] Time spent T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260504 20:01:16.408272 14818 consensus_queue.cc:1059] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260504 20:01:16.421375 14816 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:16.421756 14891 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:16.421422 14880 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:16.444972 14816 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.446053 14891 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:16.446482 14891 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:16.446877 14891 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:16.440817 14892 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:16.447256 14653 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:16.447511 14892 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:16.447956 14892 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:16.450162 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:16.450739 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.438974 14875 raft_consensus.cc:2804] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:16.452217 14875 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:16.452582 14875 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:16.454216 14899 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:16.454699 14899 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:16.455031 14899 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:16.457293 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:16.462004 12748 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:16.440335 14435 consensus_queue.cc:237] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:16.473858 14893 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.208s user 0.001s sys 0.000s
W20260504 20:01:16.486402 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:16.440091 14867 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260504 20:01:16.505244 14889 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20260504 20:01:16.486407 14876 consensus_queue.cc:237] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.506955 14752 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:16.519004 14752 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.521308 14752 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:16.510273 14867 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:16.523947 14867 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:16.527201 14816 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:16.531026 14866 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:16.532429 14605 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:16.523339 14865 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:16.541746 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.588343 12744 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:16.591337 12791 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:16.592344 14752 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:16.592818 14752 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:16.593220 14752 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:16.595731 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:16.596359 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 163614us
I20260504 20:01:16.597303 12716 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:16.598886 12716 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:16.600188 12716 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:16.596604 12749 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:16.489988 14768 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:16.605417 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 170294us
I20260504 20:01:16.506459 14900 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:16.625171 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:16.630934 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.636224 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.595908 14726 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:16.611810 12791 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:16.615352 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:16.616323 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.619661 12716 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:16.620046 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:16.622602 12750 raft_consensus.cc:1275] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:16.624101 12891 raft_consensus.cc:1275] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:16.651739 12893 raft_consensus.cc:1240] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260504 20:01:16.603749 14881 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:16.642302 12896 raft_consensus.cc:1240] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:16.601385 14887 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:16.659109 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:16.611169 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:16.638970 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.659230 12897 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260504 20:01:16.716765 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 97206.2us
I20260504 20:01:16.717768 12716 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:16.713173 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 518428us
I20260504 20:01:16.718112 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.719699 12822 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:16.659991 14900 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:16.719856 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.720293 12822 raft_consensus.cc:606] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:16.720834 12822 raft_consensus.cc:614] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:16.717073 12893 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:16.619284 14823 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:16.614691 14638 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:16.722509 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.610401 14788 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260504 20:01:16.722985 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.723675 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:16.724440 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.673024 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.724763 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.728626 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:16.728675 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 116979us
I20260504 20:01:16.731240 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.731959 12891 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:16.732959 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.733186 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:16.725347 12893 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:16.741860 14863 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:16.741285 12716 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:16.763926 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:16.768771 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 342438us
I20260504 20:01:16.787742 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:16.734551 14879 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:16.755401 14884 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260504 20:01:16.734272 14811 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:16.742197 14751 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:16.814834 12891 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:16.815572 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.734655 12896 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:16.818279 12745 raft_consensus.cc:1240] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:16.766465 14870 log.cc:927] Time spent T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
W20260504 20:01:16.850634 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 109456us
I20260504 20:01:16.851928 12716 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:16.854853 14881 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:16.855573 14881 raft_consensus.cc:697] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:16.815043 14875 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.857743 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:16.827342 12825 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:16.858695 12746 raft_consensus.cc:1240] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:16.868029 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:16.870927 12896 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:16.856576 14881 consensus_queue.cc:237] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.857374 14879 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.891201 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.892647 14879 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:16.893014 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:16.897013 14792 consensus_meta.cc:287] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.542s user 0.008s sys 0.001s
I20260504 20:01:16.868325 14811 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.899173 14900 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:16.900957 14811 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:16.857666 14751 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:16.854429 14768 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:16.901752 14751 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:16.904444 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:16.859369 14820 consensus_meta.cc:287] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.625s user 0.012s sys 0.000s
I20260504 20:01:16.843875 12741 raft_consensus.cc:1275] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260504 20:01:16.888216 14890 log.cc:927] Time spent T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260504 20:01:16.943261 14638 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:16.953429 12891 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:16.771123 14892 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:16.946832 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:16.970270 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 146581us
W20260504 20:01:16.971081 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:16.953047 14632 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:16.947989 14644 consensus_meta.cc:287] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.532s user 0.012s sys 0.000s
I20260504 20:01:16.934937 14879 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:16.859552 14899 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:16.977669 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 258593us
I20260504 20:01:16.989313 13497 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:16.975941 14724 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:16.978144 12898 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:16.997241 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:16.992046 14644 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:17.041590 12892 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260504 20:01:16.992691 14879 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:16.977007 12714 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:17.026773 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 174770us
I20260504 20:01:16.897529 14792 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:17.057001 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 211502us
W20260504 20:01:16.992499 14716 consensus_meta.cc:287] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.772s user 0.011s sys 0.002s
W20260504 20:01:17.057722 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 322132us
W20260504 20:01:17.061255 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 72046.8us
I20260504 20:01:17.062361 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.063499 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.065448 12865 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.067056 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.072391 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.084967 12822 raft_consensus.cc:2410] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:17.073451 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.083561 12819 raft_consensus.cc:606] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:17.089437 12819 raft_consensus.cc:614] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:17.090202 12823 raft_consensus.cc:1240] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260504 20:01:17.040647 14875 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:17.079185 12821 raft_consensus.cc:1240] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:17.104938 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 82875.4us
I20260504 20:01:16.990578 14632 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.068197 14544 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election won for term 6
W20260504 20:01:17.068975 14904 log.cc:927] Time spent T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.337s user 0.004s sys 0.000s
I20260504 20:01:17.068320 14716 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4.
I20260504 20:01:17.046514 14724 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.085877 14916 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:17.118222 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 141279us
I20260504 20:01:17.118885 12716 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:17.122215 12865 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.082633 14823 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:17.125839 12895 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:17.096036 14921 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:17.113549 14672 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.763s user 0.009s sys 0.004s
I20260504 20:01:17.138010 12863 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.143632 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:17.102393 14887 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:17.087049 14905 consensus_queue.cc:1048] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
W20260504 20:01:17.156680 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 74087us
I20260504 20:01:17.157589 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:17.165416 12790 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:17.183248 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 46867.5us
I20260504 20:01:17.189738 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:17.193531 12863 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.202251 12716 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:17.198046 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 57242.4us
I20260504 20:01:17.212950 12821 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:17.179818 14675 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:17.162859 14913 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260504 20:01:17.223277 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 101187us
I20260504 20:01:17.193413 14435 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:17.192754 14544 raft_consensus.cc:697] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:17.159868 14672 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.237371 12891 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:17.166231 14887 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.203882 14816 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:17.244926 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.246865 12791 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.243352 14229 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:17.250803 14927 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:17.252454 14229 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.252854 14927 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.253527 14860 raft_consensus.cc:3055] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:17.253993 14860 raft_consensus.cc:740] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:17.255021 14900 consensus_queue.cc:1048] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:01:17.255352 14927 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.256824 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:17.251016 14914 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:17.261307 12894 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:17.155225 14921 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:17.263917 14891 consensus_meta.cc:287] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.817s user 0.010s sys 0.001s
I20260504 20:01:17.264307 14921 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.264431 14891 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.265614 12899 raft_consensus.cc:1240] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
W20260504 20:01:17.265969 14880 consensus_meta.cc:287] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.816s user 0.012s sys 0.000s
I20260504 20:01:17.266484 14880 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.267048 14724 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:17.267643 14672 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.243642 14816 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.230387 14915 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:17.268641 14880 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.269703 14879 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.248592 14605 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:17.244875 14874 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:17.270781 14874 raft_consensus.cc:697] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:17.271157 14644 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:17.271749 14887 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:17.272557 14918 log.cc:927] Time spent T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260504 20:01:17.273923 14923 log.cc:927] Time spent T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260504 20:01:17.249732 14905 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:17.141036 14916 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.279275 14916 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:17.282274 14920 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260504 20:01:17.228266 14913 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:17.283989 14913 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:17.281327 14751 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.286419 14792 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.235901 14892 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.288039 14751 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:17.294441 14911 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
I20260504 20:01:17.197081 14919 consensus_queue.cc:1059] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.108s
I20260504 20:01:17.299880 14860 consensus_queue.cc:260] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.243506 14899 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.252257 14544 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.309612 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:17.306677 12894 raft_consensus.cc:1240] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:17.312958 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:17.314416 12897 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
W20260504 20:01:17.355129 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 127059us
I20260504 20:01:17.265976 14632 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:17.321561 14867 consensus_meta.cc:287] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.794s user 0.013s sys 0.001s
I20260504 20:01:17.306092 14816 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.307121 14229 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.317526 14865 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:17.246912 14924 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:17.271773 14874 consensus_queue.cc:237] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.368832 14867 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.392884 14867 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:17.374789 14865 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.395668 14895 raft_consensus.cc:3055] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:17.396325 14895 raft_consensus.cc:740] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:17.396633 14865 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.397496 14895 consensus_queue.cc:260] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.399362 14860 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:17.399134 14895 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:17.386747 14891 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.406073 14913 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.408557 14913 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:17.457388 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 254923us
I20260504 20:01:17.466540 12635 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 91ff767ed7c24b97940a459421881055 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 8)
W20260504 20:01:17.472770 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 225740us
W20260504 20:01:17.473121 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 116277us
W20260504 20:01:17.473345 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:17.482885 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 272412us
I20260504 20:01:17.485204 12896 raft_consensus.cc:1275] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:17.373947 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.499066 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:17.509099 12864 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.000s sys 0.013s
I20260504 20:01:17.511240 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:17.529219 14229 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:17.474825 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:17.538903 12864 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s user 0.000s sys 0.014s
I20260504 20:01:17.499025 14952 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260504 20:01:17.539813 14895 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:17.535856 13524 raft_consensus.cc:1240] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:17.551580 14895 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.554471 12820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:17.555783 12820 raft_consensus.cc:606] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:17.556468 12820 raft_consensus.cc:614] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260504 20:01:17.567365 14752 consensus_meta.cc:287] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.973s user 0.013s sys 0.005s
I20260504 20:01:17.570865 14653 raft_consensus.cc:3055] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:17.578177 14653 raft_consensus.cc:740] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:17.578305 14895 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.579154 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.580008 14653 consensus_queue.cc:260] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.573359 14229 consensus_queue.cc:1059] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:17.582710 14752 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.584286 14229 raft_consensus.cc:3055] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:17.584745 12825 raft_consensus.cc:1240] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260504 20:01:17.585119 14229 raft_consensus.cc:740] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:17.586915 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:17.588079 12897 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:17.588965 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 252) took 1549 ms. Trace:
I20260504 20:01:17.589520 12820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:49576
W20260504 20:01:17.589839 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 53396.8us
I20260504 20:01:17.589468 12748 rpcz_store.cc:276] 0504 20:01:16.039549 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:16.039692 (+ 143us) service_pool.cc:224] Handling call
0504 20:01:16.039932 (+ 240us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:16.462472 (+422540us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:17.588385 (+1125913us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54000dbbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:17.588749 (+ 364us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":969397120}
I20260504 20:01:17.591475 12863 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.595043 14930 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:17.595575 12715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.002s sys 0.005s
I20260504 20:01:17.595731 14752 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:17.596263 12715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.582s user 0.003s sys 0.006s
I20260504 20:01:17.595975 14930 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.597368 12892 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:17.601213 14891 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:17.602226 14891 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:17.602468 14653 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:17.602936 14891 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.575724 14916 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:17.576651 14872 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:17.574743 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 100352us
I20260504 20:01:17.604686 12646 catalog_manager.cc:5671] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 4, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:17.587787 12645 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39316 (request call id 225) took 8843 ms. Trace:
W20260504 20:01:17.558146 12716 connection.cc:254] RPC call timeout handler was delayed by -1.1715146929999491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
I20260504 20:01:17.602705 14229 consensus_queue.cc:260] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.606937 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1672) took 1119 ms. Trace:
I20260504 20:01:17.610349 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 202) took 1443 ms. Trace:
I20260504 20:01:17.610790 12897 raft_consensus.cc:1275] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:17.626385 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:17.612749 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:17.655277 12746 rpcz_store.cc:276] 0504 20:01:16.487313 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:16.487502 (+ 189us) service_pool.cc:224] Handling call
0504 20:01:16.526360 (+ 38858us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:16.859114 (+332754us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:17.606188 (+747074us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b54001ac054. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:17.606652 (+ 464us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":763831168}
I20260504 20:01:17.632647 12714 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:17.621367 14726 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.861s user 0.018s sys 0.002s
I20260504 20:01:17.637238 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:17.635737 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.615828 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.616709 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:17.633307 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:17.617888 12791 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:17.668081 12742 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:17.625193 12646 catalog_manager.cc:5671] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:17.639212 14930 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.614823 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:17.671277 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1485) took 1042 ms. Trace:
I20260504 20:01:17.664585 13529 raft_consensus.cc:2393] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:17.672091 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.672791 14863 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:17.657738 12824 rpcz_store.cc:276] 0504 20:01:16.167004 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:16.167177 (+ 173us) service_pool.cc:224] Handling call
0504 20:01:16.202015 (+ 34838us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:16.427026 (+225011us) raft_consensus.cc:1534] Early marking committed up to index 4
0504 20:01:16.626229 (+199203us) log.cc:844] Serialized 14 byte log entry
0504 20:01:16.841517 (+215288us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:16.841597 (+ 80us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:16.841666 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:17.609569 (+767903us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54000e3dd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:17.609695 (+ 126us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:17.610013 (+ 318us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52955,"prepare.queue_time_us":194774,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":59923,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":40122,"spinlock_wait_cycles":516852608,"thread_start_us":373113,"threads_started":2,"wal-append.queue_time_us":368727}
I20260504 20:01:17.630509 14952 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:17.674383 14952 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.676113 14823 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:17.677258 14952 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.677037 14823 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.608173 14872 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.679476 14653 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:17.680429 14872 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:17.680385 14653 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.680462 14895 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:17.679829 14823 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.682034 14895 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.683162 14653 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.684392 14895 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.638473 14229 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:17.686123 14952 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:17.605772 14891 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:17.671778 12742 rpcz_store.cc:276] 0504 20:01:16.628686 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:16.630561 (+ 1875us) service_pool.cc:224] Handling call
0504 20:01:17.671010 (+1040449us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002be3d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:17.671179 (+ 169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2375185152}
I20260504 20:01:17.625552 12645 rpcz_store.cc:276] 0504 20:01:08.744489 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:08.744648 (+ 159us) service_pool.cc:224] Handling call
0504 20:01:17.587433 (+8842785us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:15.355506 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:15.467029 (+111523us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:15.471059 (+ 4030us) write_op.cc:435] Acquired schema lock
0504 20:01:15.490128 (+ 19069us) tablet.cc:662] Decoding operations
0504 20:01:15.542491 (+ 52363us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:15.553682 (+ 11191us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:15.555663 (+ 1981us) tablet.cc:685] Acquiring locks for 1 operations
0504 20:01:15.566756 (+ 11093us) tablet.cc:701] Row locks acquired
0504 20:01:15.566776 (+ 20us) write_op.cc:260] PREPARE: finished
0504 20:01:15.566962 (+ 186us) write_op.cc:270] Start()
0504 20:01:15.567090 (+ 128us) write_op.cc:276] Timestamp: P: 1777924875566904 usec, L: 0
0504 20:01:15.567110 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 20:01:15.567917 (+ 807us) log.cc:844] Serialized 531 byte log entry
0504 20:01:16.004755 (+436838us) op_driver.cc:464] REPLICATION: finished
0504 20:01:16.273620 (+268865us) write_op.cc:301] APPLY: starting
0504 20:01:16.281581 (+ 7961us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:16.290300 (+ 8719us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:16.388211 (+ 97911us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:16.417181 (+ 28970us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:16.429910 (+ 12729us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 20:01:16.436501 (+ 6591us) write_op.cc:312] APPLY: finished
0504 20:01:16.457900 (+ 21399us) log.cc:844] Serialized 23 byte log entry
0504 20:01:16.458145 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:16.458251 (+ 106us) write_op.cc:454] Released schema lock
0504 20:01:16.458292 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:16.458766 (+ 474us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:16.458808 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1550944,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":254294,"apply.run_cpu_time_us":3111,"apply.run_wall_time_us":191129,"num_ops":1,"prepare.queue_time_us":542882,"prepare.run_cpu_time_us":17925,"prepare.run_wall_time_us":316128,"raft.queue_time_us":254790,"raft.run_cpu_time_us":3232,"raft.run_wall_time_us":31110,"replication_time_us":437476,"thread_start_us":421259,"threads_started":4,"wal-append.queue_time_us":179556,"wal-append.run_cpu_time_us":25742,"wal-append.run_wall_time_us":1770244}]]}
I20260504 20:01:17.691921 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:17.689397 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:17.673586 14863 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.696152 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:17.697444 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 79905.5us
I20260504 20:01:17.699241 14376 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:17.699492 12791 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:17.700177 14376 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.700688 14916 consensus_queue.cc:1059] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:17.702064 14752 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:17.702755 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.698770 14872 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:17.704290 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.703859 14376 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.706494 14544 consensus_queue.cc:1059] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:01:17.707394 14653 raft_consensus.cc:3055] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:17.707207 14930 consensus_queue.cc:1059] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:17.708034 14653 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:17.709332 14891 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:17.709412 14927 raft_consensus.cc:3055] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:17.711340 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:17.701356 14863 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:17.701442 12791 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.712090 14716 consensus_queue.cc:1059] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:17.721107 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.721451 14716 raft_consensus.cc:3055] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:17.714900 14927 raft_consensus.cc:740] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:17.721987 14716 raft_consensus.cc:740] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:17.715670 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:17.715957 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:17.713176 14865 consensus_queue.cc:1059] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:17.716521 14726 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:17.719107 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:17.725078 12819 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:17.708110 14544 raft_consensus.cc:3055] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:17.720199 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.717720 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:17.720851 14891 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:17.715780 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 83540.9us
I20260504 20:01:17.689189 12646 catalog_manager.cc:5671] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:17.716377 14952 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
W20260504 20:01:17.732961 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 104191us
I20260504 20:01:17.732820 12746 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:17.726234 14544 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:17.709685 14653 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.742756 12816 raft_consensus.cc:1240] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
W20260504 20:01:17.743191 14435 consensus_meta.cc:287] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.509s user 0.014s sys 0.008s
I20260504 20:01:17.744104 12821 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:17.730631 14726 raft_consensus.cc:697] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:01:17.744638 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:17.744872 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:17.744925 14942 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:17.746196 14768 consensus_queue.cc:1048] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:17.748148 12716 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:17.744673 12791 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.759246 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.768855 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:17.769517 14823 raft_consensus.cc:3055] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:17.769944 14872 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:17.770248 14823 raft_consensus.cc:740] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:17.770589 14872 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:17.771198 14872 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:17.771915 12716 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:17.771821 14823 consensus_queue.cc:260] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.773451 14887 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:17.762056 14950 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:17.776731 14950 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.778298 14544 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.779479 14950 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:17.779945 14544 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:17.777585 14726 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.723393 14716 consensus_queue.cc:260] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:17.782711 14716 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:17.784938 14653 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:17.786182 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.772729 14866 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:17.790079 14768 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:17.769311 14942 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.795384 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:17.796634 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:17.754128 14435 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:17.788203 14900 consensus_queue.cc:1059] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:17.723446 14927 consensus_queue.cc:260] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:17.798558 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:17.828015 12820 raft_consensus.cc:606] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.830901 12820 raft_consensus.cc:614] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:17.801170 14823 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:17.834642 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.842186 12741 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:17.842305 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:17.842502 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:17.846446 12749 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:17.846773 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:17.850711 14900 raft_consensus.cc:3055] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:17.851389 14900 raft_consensus.cc:740] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:17.851495 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260504 20:01:17.833211 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.856051 12865 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.856988 12823 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:17.857200 14807 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:17.857707 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.857246 12865 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.860666 14807 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:17.861303 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:17.867830 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 60686.1us
I20260504 20:01:17.868697 14229 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:17.878719 12865 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.879341 14768 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:17.880451 14376 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:17.880534 12865 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.881420 14376 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.882309 14902 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:17.883212 14902 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:17.884047 14376 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:17.878780 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:17.885047 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:17.886344 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:17.910787 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:17.852877 14900 consensus_queue.cc:260] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:17.914095 14900 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:17.914208 12863 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:17.877393 14807 raft_consensus.cc:2764] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260504 20:01:17.916000 14807 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:17.916441 14807 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:17.916873 14807 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:17.920204 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:01:17.877764 14952 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:17.928900 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 48502.5us
W20260504 20:01:17.943583 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:17.935530 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:17.953382 12865 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.950150 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:17.956859 12820 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:17.928481 14435 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:17.962586 12863 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:17.965281 12816 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 20:01:17.965890 12819 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:17.931087 14927 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:17.939289 14942 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:17.979241 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 234777us
I20260504 20:01:17.980743 12821 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:17.982156 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 113319us
I20260504 20:01:17.987440 12825 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260504 20:01:17.988937 12865 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:17.992033 14899 consensus_queue.cc:1059] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260504 20:01:17.994019 12716 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:17.987972 12825 pending_rounds.cc:85] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 2
W20260504 20:01:17.997365 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:17.977650 14865 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:17.978065 14902 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:18.001787 14902 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:18.002254 14902 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:17.994891 12825 pending_rounds.cc:107] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260504 20:01:18.009248 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:18.009169 12825 raft_consensus.cc:2889] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260504 20:01:18.014439 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 49551.2us
I20260504 20:01:18.015568 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:18.027765 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:18.010278 14891 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:18.031563 12863 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:18.035758 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:18.019227 14768 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:18.054607 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 65748.8us
W20260504 20:01:18.057432 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 45300.6us
I20260504 20:01:18.034806 14653 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:18.060375 14229 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:18.060827 14229 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:18.061126 14229 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:18.026038 14895 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:18.063791 12820 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:18.073776 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:18.067286 14376 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:18.078688 14376 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:18.081039 14376 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:18.081722 12863 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:18.082495 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:18.083047 14942 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:18.083675 14942 raft_consensus.cc:697] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:18.083745 12865 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:18.084923 14942 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:18.082548 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 75007.5us
I20260504 20:01:18.090050 12865 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:18.099344 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 116496us
I20260504 20:01:18.094885 14891 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:18.095374 14895 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:18.114864 12791 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:18.101842 14899 raft_consensus.cc:2804] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:18.090175 14653 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:18.132045 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:18.134727 12716 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.136602 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:18.137558 12819 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:18.130195 14376 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260504 20:01:18.154785 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.156499 14913 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:18.160629 14891 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:18.140439 14899 raft_consensus.cc:697] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:18.161281 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:18.182449 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 77781.5us
W20260504 20:01:18.182868 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 55998.3us
I20260504 20:01:18.155273 14376 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:18.161389 14891 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.188203 12716 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:18.192101 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.167143 14895 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:18.196770 14229 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:18.208673 14962 log.cc:927] Time spent T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260504 20:01:18.189404 14376 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:18.213982 12816 raft_consensus.cc:1275] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260504 20:01:18.147907 14952 consensus_queue.cc:1048] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260504 20:01:18.216430 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:18.216964 12816 pending_rounds.cc:85] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 1
I20260504 20:01:18.217204 14229 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:18.217368 12816 pending_rounds.cc:107] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260504 20:01:18.217828 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.226184 12816 raft_consensus.cc:2889] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:18.218261 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.183786 14902 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.222726 14895 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:18.249055 14860 consensus_meta.cc:287] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.712s user 0.018s sys 0.006s
I20260504 20:01:18.175899 14807 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.264886 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:18.268052 12820 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:18.260821 12746 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:18.255713 14952 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260504 20:01:18.294548 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 110293us
I20260504 20:01:18.302369 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:18.268417 14716 raft_consensus.cc:3055] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:18.324947 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:18.326457 14960 log.cc:927] Time spent T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260504 20:01:18.327656 14891 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:18.331373 14807 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:18.332772 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.335394 14895 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:18.333334 14899 consensus_queue.cc:237] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:18.336606 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:18.285194 14768 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:18.321583 14653 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:18.338845 14716 raft_consensus.cc:740] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:18.384023 14966 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:18.387960 14966 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.392094 14902 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:18.416630 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.449864 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.450256 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.451192 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:18.452693 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:18.453594 12821 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260504 20:01:18.384742 14768 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:18.464084 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:18.468233 12825 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:18.468842 12743 raft_consensus.cc:2393] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:18.471662 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:18.472004 12742 raft_consensus.cc:2393] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:18.472074 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:18.473004 12821 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:18.473021 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.474566 12897 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:18.476491 12745 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260504 20:01:18.479940 12863 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:18.479382 12895 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:18.458000 14865 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:18.490137 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 226535us
I20260504 20:01:18.488082 12819 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260504 20:01:18.493683 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.500988 12820 raft_consensus.cc:2393] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:18.502238 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.492161 14902 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:18.522709 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.482314 14435 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20260504 20:01:18.530990 12895 raft_consensus.cc:2410] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:18.534137 13532 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260504 20:01:18.534438 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.539562 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:18.521744 14902 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:18.549935 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 92321us
I20260504 20:01:18.550701 12898 raft_consensus.cc:2376] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4. Re-sending same reply.
I20260504 20:01:18.512646 14865 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.551654 12714 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:18.543704 14726 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260504 20:01:18.561928 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:18.562740 12822 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:18.562909 14768 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:18.555734 14966 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:18.555050 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 185803us
I20260504 20:01:18.568109 12821 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:18.546501 14902 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:18.576118 12863 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:18.548944 14895 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:18.574810 12864 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.582577 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:18.566035 14544 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.786s user 0.020s sys 0.002s
I20260504 20:01:18.584389 14860 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:18.584066 14895 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.584971 14860 raft_consensus.cc:697] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:18.586071 14860 consensus_queue.cc:237] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.586583 14895 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:18.588707 14768 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:18.596412 12864 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.598271 14952 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:18.598469 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 47006.4us
I20260504 20:01:18.598963 14952 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.598146 14967 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:18.600615 14967 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:18.600785 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:18.602274 14974 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:18.603044 14974 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:18.604431 14872 consensus_meta.cc:287] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.833s user 0.012s sys 0.004s
I20260504 20:01:18.604960 14872 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.606480 14974 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:18.607607 14865 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:18.608075 14974 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:18.608755 14974 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:18.607152 14768 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:18.597990 14935 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
W20260504 20:01:18.612558 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.600826 12895 raft_consensus.cc:1275] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:18.601542 12748 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:18.632623 12823 raft_consensus.cc:1240] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:18.636266 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:18.631088 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 47410.2us
W20260504 20:01:18.633925 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.639492 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:18.640045 14768 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.629479 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:18.641832 14435 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:18.642329 14768 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:18.640487 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:18.643221 12820 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
W20260504 20:01:18.638891 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 48173.5us
I20260504 20:01:18.645354 12863 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:18.658155 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:18.642974 14768 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:18.645092 14952 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:18.654575 14968 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:01:18.676784 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.669893 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:18.654843 14974 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:18.685299 14229 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:18.690071 14229 raft_consensus.cc:697] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:18.690689 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 91100.5us
I20260504 20:01:18.656663 14376 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.679894 14968 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:18.694336 12865 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.675490 14768 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:18.698160 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 44999.4us
I20260504 20:01:18.699699 12714 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:18.699363 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:18.710812 14927 consensus_meta.cc:287] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.732s user 0.015s sys 0.003s
I20260504 20:01:18.710638 14768 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:18.699084 14968 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:18.698904 14967 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:18.715968 12822 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:18.720207 12863 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:18.719883 14980 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:18.722445 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.721374 14980 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.727499 14872 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:18.728438 14980 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:18.729326 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.729902 12791 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.730240 12744 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:18.732417 12791 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:18.734159 12749 raft_consensus.cc:2393] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:18.734090 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:18.736717 14653 consensus_queue.cc:1059] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:18.736843 12716 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:18.737252 14981 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:18.733839 14229 consensus_queue.cc:237] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.715842 14895 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:18.743201 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.745149 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.745623 14979 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:18.746647 14979 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:18.747010 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.747903 12893 raft_consensus.cc:2393] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:18.748854 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.749456 14716 consensus_queue.cc:260] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.753432 12716 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.754516 14982 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:18.754974 14969 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:18.754305 14872 consensus_queue.cc:1059] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260504 20:01:18.759416 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 64703.6us
I20260504 20:01:18.760499 12714 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.760924 12865 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:18.770143 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:18.747063 14979 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:18.771373 14979 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:18.771587 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.774806 14980 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:18.775346 14980 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:18.775219 14902 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.775843 14980 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:18.749511 12893 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:18.778755 14376 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:18.778520 14968 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.779470 14971 raft_consensus.cc:3055] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:18.779986 14971 raft_consensus.cc:740] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:18.780954 14865 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:18.781553 14865 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:18.781015 14971 consensus_queue.cc:260] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.782073 14865 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:18.782362 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:18.782821 14971 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:18.783059 12823 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:18.786784 14969 raft_consensus.cc:697] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:18.788221 14969 consensus_queue.cc:237] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:18.789053 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:18.790599 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:18.798492 12748 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:18.800122 12823 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:18.801920 14899 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:18.738692 14768 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260504 20:01:18.805866 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 45482.9us
I20260504 20:01:18.805620 12863 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:18.807616 14726 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:18.813117 14726 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:18.815765 14967 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:18.816452 14967 raft_consensus.cc:697] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:18.817487 14967 consensus_queue.cc:237] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:18.819204 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 59767.3us
I20260504 20:01:18.820047 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:18.800774 14968 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:18.823360 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.826634 14435 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:18.827111 14435 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:18.827553 14435 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:18.830183 12890 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:18.839138 14887 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
W20260504 20:01:18.849197 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.852808 14974 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:18.813655 14726 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:18.827272 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 121755us
I20260504 20:01:18.879372 12748 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:18.879556 12716 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:18.858417 14887 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
W20260504 20:01:18.908962 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 52408.1us
I20260504 20:01:18.905328 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:18.926005 12865 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:18.927042 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 194742us
I20260504 20:01:18.925823 12714 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:18.944613 12820 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:18.903085 14902 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:18.883821 14981 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260504 20:01:18.944659 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:18.971401 12748 raft_consensus.cc:2393] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:18.923166 14891 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:18.902045 14807 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:18.976195 14229 consensus_queue.cc:1059] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260504 20:01:18.983479 12823 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:18.987804 14229 raft_consensus.cc:3055] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:18.989929 14229 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:18.976574 14653 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:18.985036 14993 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:19.000149 14229 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.001854 14229 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:19.002743 12791 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:19.009097 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:19.001083 14979 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.021612 14982 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:19.026557 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 100865us
I20260504 20:01:19.029145 15003 consensus_queue.cc:1059] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:19.031667 15003 raft_consensus.cc:3055] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:19.032178 15003 raft_consensus.cc:740] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:19.032483 14544 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260504 20:01:18.985167 14807 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:19.034651 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 153031us
I20260504 20:01:19.005997 14891 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:19.036810 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:18.993475 14823 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:19.040042 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.037635 12716 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:18.980703 14860 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260504 20:01:19.041657 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.024330 14716 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:19.044816 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.033710 14895 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:19.044133 12899 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:19.056146 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.058198 12716 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:19.059432 14913 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:19.044958 14995 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:19.063051 14995 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:19.063570 14995 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:19.064258 14435 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:19.065222 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:19.067066 14979 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:19.067828 14952 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:19.068014 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.068707 12895 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:19.068861 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.069471 12716 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:19.070210 14435 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:19.062546 14980 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.070984 14952 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:19.072253 14980 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:19.062501 14913 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:19.074162 14913 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:19.074201 14952 consensus_queue.cc:1059] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:19.075193 14807 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:19.079504 14865 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.081138 12894 raft_consensus.cc:2393] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 2: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:19.082144 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.083127 12890 raft_consensus.cc:2393] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:19.085335 14891 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:19.087714 13530 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:19.088917 14726 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:19.095381 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:19.096989 14952 consensus_queue.cc:1243] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer 732c3560a688494196d22a1a48f89968 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260504 20:01:19.097709 14872 raft_consensus.cc:3055] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:19.098241 14872 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:19.099231 14872 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.100698 14872 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:19.022152 14653 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:19.103374 14653 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:19.105193 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:19.054808 14823 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.106235 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:19.106957 12824 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:19.107268 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:19.107806 14823 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:19.107990 13532 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:19.108178 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:19.108973 12819 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:19.109089 14726 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:19.109978 12790 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:19.110975 14229 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:19.111337 14229 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:19.111663 14229 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:19.112764 12633 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 0e8eba4dbb8d436f81062ca3bbf46788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 5ff0ef72ec3b4667893a0420d3c7f923 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:19.114411 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:19.115088 14952 raft_consensus.cc:3055] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:19.115711 14952 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:19.115862 12892 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 20:01:19.116645 14952 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:19.118180 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:19.119496 12823 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:19.121135 14899 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:19.042704 15003 consensus_queue.cc:260] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.123200 14979 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:19.123850 14979 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.126416 14979 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:19.134421 13527 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:19.135236 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40940 (request call id 260) took 1094 ms. Trace:
I20260504 20:01:19.135757 14229 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:19.135947 13527 rpcz_store.cc:276] 0504 20:01:18.041302 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:18.042037 (+ 735us) service_pool.cc:224] Handling call
0504 20:01:18.582508 (+540471us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:18.788875 (+206367us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d39ec 00007f01598d35ab 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:19.135136 (+346261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30732544}
I20260504 20:01:19.138239 14229 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:19.139144 15003 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:19.142616 14952 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:19.143956 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40940 (request call id 261) took 1071 ms. Trace:
I20260504 20:01:19.144570 13530 rpcz_store.cc:276] 0504 20:01:18.073260 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:18.073426 (+ 166us) service_pool.cc:224] Handling call
0504 20:01:19.143712 (+1070286us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b54006ef354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:19.143867 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1481601408}
I20260504 20:01:19.115154 12899 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:19.147486 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:19.135548 15002 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.105s user 0.000s sys 0.003s
I20260504 20:01:19.151217 14998 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:19.151692 12741 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:19.151911 14998 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:19.153694 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:19.176615 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:19.195806 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:19.210964 12714 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:19.215269 12820 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260504 20:01:19.215781 12897 raft_consensus.cc:1240] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:19.193324 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:19.219704 12744 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260504 20:01:19.220435 12714 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:19.227222 12714 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:19.228335 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.229473 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:19.230119 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.244335 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:19.251485 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.252985 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.206269 14980 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260504 20:01:19.224560 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:19.252180 12743 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:19.250476 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:19.222646 14435 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:19.251516 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:19.222961 14544 consensus_queue.cc:1059] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260504 20:01:19.260102 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:19.357474 12744 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:19.336932 12823 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:19.325188 12899 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:19.314587 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:19.270155 12898 raft_consensus.cc:1240] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:19.230154 14974 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:19.362284 12895 raft_consensus.cc:2410] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:19.282464 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:19.366751 12820 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:19.366793 12891 raft_consensus.cc:2410] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260504 20:01:19.314996 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 51285.1us
I20260504 20:01:19.324860 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:19.307248 12863 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:19.232230 14981 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:19.374851 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.376183 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.382447 12750 raft_consensus.cc:1240] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:19.384641 12749 raft_consensus.cc:1240] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:19.251303 14887 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:19.233253 14865 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:19.384574 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 77472.9us
I20260504 20:01:19.391855 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:19.392815 12823 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:19.394817 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.365991 13527 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:19.400722 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:19.401647 12743 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:19.406260 14807 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:19.406991 14807 raft_consensus.cc:697] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:19.408000 14807 consensus_queue.cc:237] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.409997 12747 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:19.410176 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:19.365772 14974 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:19.414079 14974 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:19.414623 12821 raft_consensus.cc:1240] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:19.328466 14998 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:19.380496 14981 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:19.428659 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:19.437451 12716 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:19.397913 14982 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:19.448143 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.448251 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.449141 12742 raft_consensus.cc:2393] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 7: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:19.450282 12747 raft_consensus.cc:2393] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:19.423656 14860 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:19.472977 12823 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
W20260504 20:01:19.473688 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 83452.7us
I20260504 20:01:19.480340 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.437036 14981 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:19.483395 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 56863.1us
I20260504 20:01:19.399747 14435 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:19.448128 14865 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:19.450960 14966 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:19.497578 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:19.492570 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:19.497244 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:19.505525 12824 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:19.452046 15006 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:19.511623 12821 raft_consensus.cc:1275] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:19.512498 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:19.505184 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:19.503891 12816 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:19.520313 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.488030 14967 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:19.533509 15006 raft_consensus.cc:697] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:19.534780 12863 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:19.535089 15006 consensus_queue.cc:237] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.539736 15008 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:19.567775 12819 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:19.468389 14860 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.566862 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:19.504159 14982 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:19.576767 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:19.516733 14865 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:19.576987 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 67973.2us
I20260504 20:01:19.662936 14435 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:19.655061 14900 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
W20260504 20:01:19.562410 13497 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.165s (SENT) [suppressed 6 similar messages]
I20260504 20:01:19.632566 12820 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:19.628159 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:19.693615 12865 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:19.569278 14895 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:19.665992 14995 consensus_meta.cc:287] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.602s user 0.010s sys 0.003s
I20260504 20:01:19.562049 14969 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260504 20:01:19.578366 14913 consensus_meta.cc:287] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.503s user 0.013s sys 0.000s
I20260504 20:01:19.661782 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:19.657238 14229 consensus_queue.cc:1059] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260504 20:01:19.749606 12822 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:19.759047 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 46026.3us
I20260504 20:01:19.541740 15008 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.193s
I20260504 20:01:19.780936 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:19.578158 14967 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.742734 13527 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:19.556399 14966 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.567329 15015 consensus_queue.cc:1059] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.200s
W20260504 20:01:19.760234 14952 consensus_meta.cc:287] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.617s user 0.007s sys 0.006s
I20260504 20:01:19.820050 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:19.801703 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 58129.6us
W20260504 20:01:19.811731 15011 log.cc:927] Time spent T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.076s user 0.000s sys 0.002s
W20260504 20:01:19.839305 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:19.814230 14895 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:19.820832 14913 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:19.842564 14762 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:19.843333 14913 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:19.847878 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:19.848268 15016 log.cc:927] Time spent T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20260504 20:01:19.848867 12820 raft_consensus.cc:2368] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 3. Current term is 4.
I20260504 20:01:19.849995 12820 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:19.809324 14969 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:19.864377 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 301983us
W20260504 20:01:19.865929 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 124865us
I20260504 20:01:19.910753 12790 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:19.915550 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 203538us
I20260504 20:01:19.837255 14891 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:19.908444 14895 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:19.843392 14762 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:19.983116 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:19.983639 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:19.861794 14969 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:19.940560 14900 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:19.953609 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 106844us
I20260504 20:01:19.956039 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:19.966648 12744 consensus_meta.cc:287] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.609s user 0.015s sys 0.000s
I20260504 20:01:20.000947 12744 raft_consensus.cc:2410] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:19.966215 15033 consensus_queue.cc:1059] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.115s
W20260504 20:01:19.967840 15019 log.cc:927] Time spent T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20260504 20:01:19.721374 14982 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:19.986528 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 166475us
I20260504 20:01:19.814787 14995 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.021175 12817 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:20.021433 14995 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.022023 14762 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.019323 14994 raft_consensus.cc:3055] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Stepping down as leader of term 4
W20260504 20:01:19.995968 14716 consensus_meta.cc:287] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.937s user 0.010s sys 0.008s
I20260504 20:01:19.855710 14952 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:20.039791 14952 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.022886 14994 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:20.081132 12744 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260504 20:01:20.081772 12744 pending_rounds.cc:85] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting all ops after (but not including) 3
I20260504 20:01:20.082101 12744 pending_rounds.cc:107] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260504 20:01:20.082402 12744 raft_consensus.cc:2889] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:20.017930 12896 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:20.089275 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.090991 14860 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.092880 12750 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:20.092666 14994 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.095398 12822 raft_consensus.cc:1240] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:20.095834 12714 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:20.130082 12743 consensus_meta.cc:287] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.712s user 0.009s sys 0.005s
I20260504 20:01:20.130939 12743 raft_consensus.cc:2410] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:20.135816 12818 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 6. Current term is 7. Ops: []
W20260504 20:01:20.136454 15029 log.cc:927] Time spent T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20260504 20:01:20.058125 14865 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.077904 14967 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.140022 14966 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.094635 15006 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:20.140976 15006 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.143157 15006 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.132592 15007 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:20.144330 15007 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.144778 12750 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:20.078207 14435 consensus_queue.cc:1059] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
W20260504 20:01:20.147516 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 51750.4us
I20260504 20:01:19.959556 14981 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.158751 14435 raft_consensus.cc:3055] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:20.159296 14435 raft_consensus.cc:740] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:20.160218 14435 consensus_queue.cc:260] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.160974 13529 raft_consensus.cc:1275] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:20.058511 14900 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.163857 12714 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:20.164595 14900 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:20.166442 15034 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:20.161599 14435 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:20.152374 15003 consensus_meta.cc:287] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.013s user 0.013s sys 0.001s
I20260504 20:01:20.156262 15035 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:20.244305 14807 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:20.244544 15003 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:20.247730 14807 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.247581 15003 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.252799 14900 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:20.253194 14930 raft_consensus.cc:3055] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:20.253615 14930 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:20.253463 14900 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.254603 14952 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.254598 14930 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.256013 14981 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.256274 14930 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:20.257261 14900 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:20.259868 15003 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.173045 14982 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:20.263036 14887 consensus_meta.cc:287] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.872s user 0.010s sys 0.003s
I20260504 20:01:20.263568 14887 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:20.266953 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1615) took 1052 ms. Trace:
I20260504 20:01:20.267378 12896 rpcz_store.cc:276] 0504 20:01:19.214955 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:19.217613 (+ 2658us) service_pool.cc:224] Handling call
0504 20:01:19.955807 (+738194us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:20.266733 (+310926us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b54001b8d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:20.266856 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55768704}
I20260504 20:01:20.268551 14900 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:20.269907 13524 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:20.269558 14900 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.272470 14792 raft_consensus.cc:3055] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:20.272922 14792 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:20.273128 14900 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:19.857906 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:20.187152 15034 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:20.276700 15034 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.282729 15034 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:20.287570 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 594059us
I20260504 20:01:20.289355 12865 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:20.328173 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 148746us
I20260504 20:01:20.330422 12746 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
W20260504 20:01:20.337822 14974 consensus_meta.cc:287] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.923s user 0.015s sys 0.001s
I20260504 20:01:20.337285 14900 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:20.391899 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.392175 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.392649 12750 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:20.398859 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.400004 12743 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:20.332044 15035 raft_consensus.cc:697] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:20.342542 14895 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260504 20:01:20.274063 14792 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.405125 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.409502 12742 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:20.410569 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:20.411250 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:20.411278 12750 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:20.390527 14900 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:20.386698 14891 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260504 20:01:20.366336 14860 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260504 20:01:20.365032 15025 consensus_queue.cc:1048] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260504 20:01:20.449306 12749 raft_consensus.cc:1240] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:20.440338 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 529678us
I20260504 20:01:20.433665 14979 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:20.441567 14995 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:20.467921 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1530) took 1142 ms. Trace:
W20260504 20:01:20.421887 12713 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.000s sys 0.030s
W20260504 20:01:20.468713 12713 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.000s sys 0.030s
I20260504 20:01:20.468482 12746 rpcz_store.cc:276] 0504 20:01:19.306368 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:19.306520 (+ 152us) service_pool.cc:224] Handling call
0504 20:01:20.424052 (+1117532us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b54002e4fd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:20.448902 (+ 24850us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2078103808}
I20260504 20:01:20.441872 14865 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:20.444150 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:20.470378 14865 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.435691 14229 raft_consensus.cc:3055] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:20.471438 14229 raft_consensus.cc:740] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:20.472388 14229 consensus_queue.cc:260] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.472855 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:20.417100 14900 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:20.445711 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:20.447788 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:20.441704 15037 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:20.416683 14966 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:20.414276 14807 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:20.473793 13524 raft_consensus.cc:2368] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:20.429338 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.483278 15007 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.483868 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.485011 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:20.485244 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.485558 12821 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 3
I20260504 20:01:20.421284 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.422308 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:20.420847 12818 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:20.422307 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.417680 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.422644 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.498667 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.429598 14792 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:20.447099 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:20.442287 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:20.487905 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.474846 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.478214 15035 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.476183 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:20.505329 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.507826 12820 raft_consensus.cc:2393] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:20.475425 14229 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:20.479200 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.485956 12747 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:20.502254 12743 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:20.472774 14865 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.503432 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:20.487440 15043 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:20.481349 15037 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.513849 15043 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.516271 15043 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.481645 14966 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.391045 14974 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.506167 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.486254 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:20.525298 12818 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:20.516299 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:20.524555 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 82360.5us
W20260504 20:01:20.529739 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:20.538703 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.539467 12820 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:20.540537 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.542093 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:20.541422 12818 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:20.549882 12715 leader_election.cc:400] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:20.550266 12715 leader_election.cc:403] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:20.552083 12714 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:20.554653 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:20.555382 12747 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 4
I20260504 20:01:20.554566 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.556622 12714 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:20.441789 14969 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.564875 12825 raft_consensus.cc:2410] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:20.567860 12744 raft_consensus.cc:2410] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:20.570170 12820 raft_consensus.cc:3055] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:20.570554 12820 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:20.572525 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.573225 12818 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:20.571516 12820 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.468205 14979 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.578816 12745 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 6. Current term is 7. Ops: []
W20260504 20:01:20.579444 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 76144.4us
I20260504 20:01:20.580228 12791 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:20.505151 14995 raft_consensus.cc:697] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:20.576674 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:20.577615 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.583648 12822 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:20.548914 15003 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:20.584095 12816 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:20.585523 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:20.574749 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:20.592140 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:20.574316 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.587886 12820 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:20.548605 14966 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.598122 12822 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:20.577232 14768 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:20.606727 14768 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:20.607250 14768 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:20.610476 14995 consensus_queue.cc:237] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.611209 12741 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:20.613231 14865 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:20.614102 14865 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.615104 15001 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:20.615614 15001 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:20.616181 15001 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:20.616653 15003 raft_consensus.cc:697] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:20.618176 15049 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:20.618762 15049 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:20.619267 15049 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:20.605240 15048 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:20.622012 15048 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.624693 15048 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.625737 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.626506 12825 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:20.628865 15003 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:20.569893 15034 consensus_queue.cc:1059] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260504 20:01:20.631158 15034 raft_consensus.cc:3055] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:20.631289 12714 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:20.631610 15034 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:20.632313 12714 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:20.632525 15034 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.615545 15008 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:20.577003 15041 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:20.619333 15037 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.642371 15041 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.645359 15041 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.647138 14974 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:20.647390 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1720) took 1547 ms. Trace:
I20260504 20:01:20.659832 12817 rpcz_store.cc:276] 0504 20:01:19.099702 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:19.099875 (+ 173us) service_pool.cc:224] Handling call
0504 20:01:19.100307 (+ 432us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:20.021586 (+921279us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:20.646968 (+625382us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b5400374454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:20.647277 (+ 309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2116894080}
I20260504 20:01:20.648902 14900 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.654404 14865 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:20.658344 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:20.669898 12820 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260504 20:01:20.676641 12820 pending_rounds.cc:85] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 2
I20260504 20:01:20.677021 12820 pending_rounds.cc:107] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260504 20:01:20.677317 12820 raft_consensus.cc:2889] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:20.688964 12747 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:20.692050 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1695) took 1302 ms. Trace:
I20260504 20:01:20.692612 12749 rpcz_store.cc:276] 0504 20:01:19.390176 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:19.390356 (+ 180us) service_pool.cc:224] Handling call
0504 20:01:20.449035 (+1058679us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:20.466187 (+ 17152us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:20.688810 (+222623us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002e4fd0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:20.691960 (+ 3150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2433915776}
I20260504 20:01:20.700158 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:20.701087 12896 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:20.695546 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.695395 15022 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
W20260504 20:01:20.638104 15033 consensus_queue.cc:1175] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "732c3560a688494196d22a1a48f89968" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 4 index: 3 } }
I20260504 20:01:20.709048 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:20.709797 12897 raft_consensus.cc:2393] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:20.709717 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.710525 12894 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:20.710745 14969 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.711367 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:20.712114 12894 raft_consensus.cc:2393] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:20.713578 14969 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:20.714075 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.714208 14969 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.714814 12890 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:20.715801 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.716253 14969 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.716466 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.716961 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.717047 12865 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:20.718101 12865 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:20.718870 12892 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260504 20:01:20.719137 14865 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:20.719190 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:20.719743 14865 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:20.720155 14865 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:20.720204 12865 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:20.721159 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:20.721978 15037 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:20.722378 15037 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:20.722728 15037 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:20.722936 15050 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:20.723348 15050 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:20.666189 15008 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.672295 15035 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:20.724743 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:20.664091 14993 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:20.710760 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.739532 12897 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:20.735533 12821 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:20.634747 14872 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:20.671945 15034 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:20.741747 12865 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:20.743121 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:20.747637 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.752513 12865 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:20.753113 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.754033 12749 raft_consensus.cc:1275] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:20.754066 12821 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:20.730703 14900 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:20.723673 15050 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:20.755340 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.757979 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:20.732430 15048 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:20.765300 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:20.772251 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 192103us
I20260504 20:01:20.773979 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.774583 12891 raft_consensus.cc:1240] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:20.774761 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.717733 12890 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:20.775341 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:20.742556 15035 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:20.776835 12646 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39338 (request call id 228) took 12109 ms (client timeout 15000 ms). Trace:
I20260504 20:01:20.742084 14823 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:20.742274 14942 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:20.752264 14980 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:20.755770 14966 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:20.790302 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:20.804899 12743 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:20.762189 15009 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:20.783139 12895 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:20.762503 14716 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:20.799706 12716 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:20.802929 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:20.806923 12749 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:20.808391 12897 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:20.812003 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:20.812820 12750 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:20.821691 12791 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:20.824749 12716 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:20.821177 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:20.791476 15035 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:20.837383 14998 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:20.776163 15048 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.820006 14823 raft_consensus.cc:697] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:20.889919 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 74222.6us
I20260504 20:01:20.843504 15001 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:20.850597 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 60785.1us
I20260504 20:01:20.853178 12716 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:20.857918 12750 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
W20260504 20:01:20.787714 12646 rpcz_store.cc:269] 0504 20:01:08.667649 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:08.667828 (+ 179us) service_pool.cc:224] Handling call
0504 20:01:20.776732 (+12108904us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:18.434456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:18.435228 (+ 772us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:18.435299 (+ 71us) write_op.cc:435] Acquired schema lock
0504 20:01:18.435317 (+ 18us) tablet.cc:662] Decoding operations
0504 20:01:18.435966 (+ 649us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:18.436054 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:18.436082 (+ 28us) tablet.cc:685] Acquiring locks for 3 operations
0504 20:01:18.436692 (+ 610us) tablet.cc:701] Row locks acquired
0504 20:01:18.436723 (+ 31us) write_op.cc:260] PREPARE: finished
0504 20:01:18.436913 (+ 190us) write_op.cc:270] Start()
0504 20:01:18.437073 (+ 160us) write_op.cc:276] Timestamp: P: 1777924878436843 usec, L: 0
0504 20:01:18.437112 (+ 39us) op_driver.cc:348] REPLICATION: starting
0504 20:01:18.476158 (+ 39046us) log.cc:844] Serialized 1294 byte log entry
0504 20:01:18.922670 (+446512us) op_driver.cc:464] REPLICATION: finished
0504 20:01:19.402770 (+480100us) write_op.cc:301] APPLY: starting
0504 20:01:19.402935 (+ 165us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:19.403140 (+ 205us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:19.403314 (+ 174us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:19.404046 (+ 732us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:19.404400 (+ 354us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0504 20:01:19.404494 (+ 94us) write_op.cc:312] APPLY: finished
0504 20:01:19.404821 (+ 327us) log.cc:844] Serialized 35 byte log entry
0504 20:01:19.405054 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:19.405164 (+ 110us) write_op.cc:454] Released schema lock
0504 20:01:19.405227 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:19.405718 (+ 491us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:19.405746 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":188,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":438366,"apply.run_cpu_time_us":3081,"apply.run_wall_time_us":27628,"num_ops":3,"prepare.queue_time_us":508819,"prepare.run_cpu_time_us":13314,"prepare.run_wall_time_us":200460,"raft.queue_time_us":164228,"raft.run_cpu_time_us":28235,"raft.run_wall_time_us":308586,"replication_time_us":461125,"thread_start_us":617054,"threads_started":4,"wal-append.queue_time_us":171655,"wal-append.run_cpu_time_us":15011,"wal-append.run_wall_time_us":1865286}]]}
I20260504 20:01:20.823840 14993 raft_consensus.cc:697] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:01:20.943364 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 90307us
I20260504 20:01:20.946892 12743 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:20.931193 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:20.951972 12896 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:20.807502 14980 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:20.812000 15043 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:20.850311 15054 raft_consensus.cc:2764] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20260504 20:01:20.992272 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:20.984504 12817 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:21.005090 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 156724us
I20260504 20:01:21.014604 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:21.027274 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:20.967957 14981 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:21.049386 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 116322us
I20260504 20:01:20.990005 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:20.930424 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 68941us
I20260504 20:01:20.951160 14979 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:20.859019 14998 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:20.898139 14969 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:21.098039 14998 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:21.087121 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.102888 12822 raft_consensus.cc:2393] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:21.061214 15025 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:21.097680 12865 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.062650 14807 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:21.117409 12865 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.141204 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:21.055820 14967 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260504 20:01:21.110366 14979 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:21.126629 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 195986us
I20260504 20:01:21.151902 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:21.152364 14980 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:21.123533 14900 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:21.154350 14900 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:21.154750 14900 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:21.153556 12791 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:21.157166 15054 consensus_queue.cc:1059] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:21.158211 14229 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:21.158561 15054 raft_consensus.cc:3055] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:21.159039 15054 raft_consensus.cc:740] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:21.159945 15054 consensus_queue.cc:260] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.161440 15054 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:21.133910 14807 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.164841 14807 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:21.125092 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:21.077391 15043 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:21.118849 15049 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.144663 12819 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:21.119220 15032 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:21.142537 15008 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:21.145157 14891 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
W20260504 20:01:21.148695 14792 consensus_meta.cc:287] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.625s user 0.012s sys 0.000s
W20260504 20:01:21.150887 14768 consensus_meta.cc:287] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.543s user 0.011s sys 0.001s
I20260504 20:01:21.164037 12647 catalog_manager.cc:5671] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 2 to 3, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 3 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:01:21.141935 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:21.167757 12865 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.172353 14229 consensus_queue.cc:1059] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:01:21.104499 14969 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:21.177338 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:21.179044 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:21.194509 12647 catalog_manager.cc:5671] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:21.197247 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.197436 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:21.199144 12822 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:21.203425 12791 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:21.207194 12819 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:21.186004 15008 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:21.220361 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.189754 15038 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:21.185845 14891 raft_consensus.cc:697] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:21.231132 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:21.167936 15025 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.170260 14993 consensus_queue.cc:237] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.184024 14768 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.242426 12819 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:21.258914 12863 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.245345 12714 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.224221 13524 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260504 20:01:21.226702 15001 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:21.180573 14823 consensus_queue.cc:237] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.274323 12818 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:21.276454 12818 raft_consensus.cc:606] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:21.276826 12818 raft_consensus.cc:614] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:21.276001 12647 catalog_manager.cc:5671] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 2 to 3, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 3 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:21.274608 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.274356 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.278457 12819 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:21.278496 12822 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
W20260504 20:01:21.256574 15056 log.cc:927] Time spent T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260504 20:01:21.244007 15048 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:21.287969 12647 catalog_manager.cc:5671] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:21.287922 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:21.268282 15058 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.344s user 0.002s sys 0.000s
I20260504 20:01:21.247434 15008 raft_consensus.cc:697] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:21.273662 15059 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.342s user 0.002s sys 0.000s
W20260504 20:01:21.296438 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 93274.1us
I20260504 20:01:21.312407 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:21.323766 12742 raft_consensus.cc:1240] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:21.325649 12823 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260504 20:01:21.333823 14887 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:21.334188 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.334900 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:21.346846 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 101582us
I20260504 20:01:21.336879 12647 catalog_manager.cc:5671] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:01:21.338550 15060 log.cc:927] Time spent T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20260504 20:01:21.244139 14980 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260504 20:01:21.247151 15041 consensus_queue.cc:1059] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
W20260504 20:01:21.280441 14865 consensus_meta.cc:287] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.560s user 0.015s sys 0.000s
I20260504 20:01:21.272228 14967 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:21.360657 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:21.343725 12747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.009s sys 0.017s
W20260504 20:01:21.361632 12747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.010s sys 0.018s
I20260504 20:01:21.348106 12817 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:21.335669 13525 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260504 20:01:21.299705 12741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1702) took 1654 ms (client timeout 1534 ms). Trace:
W20260504 20:01:21.358490 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 414197us
I20260504 20:01:21.308434 15072 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:21.368116 12741 rpcz_store.cc:269] 0504 20:01:19.634759 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:19.636690 (+ 1931us) service_pool.cc:224] Handling call
0504 20:01:21.272020 (+1635330us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b540022f1d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:21.288883 (+ 16863us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":872138624}
I20260504 20:01:21.368417 15072 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.341923 14716 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:21.374008 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:21.244467 15038 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.385689 12791 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.364482 14887 raft_consensus.cc:697] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:21.352231 15049 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:21.391563 12818 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:21.395612 12818 raft_consensus.cc:606] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:21.396157 12818 raft_consensus.cc:614] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:21.370985 12647 catalog_manager.cc:5671] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 2 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:01:21.309163 14891 consensus_queue.cc:237] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.408785 12863 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:21.414321 12822 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:21.390216 14768 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:21.387180 15041 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:21.418900 12863 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.419636 12791 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.364552 14865 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:21.445515 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:21.407704 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:21.409659 14979 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:21.443437 12647 catalog_manager.cc:5671] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 1 to 2, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 2 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:21.388059 14716 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.454365 12716 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.386718 15048 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260504 20:01:21.455448 15073 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:21.457736 15072 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:21.458058 15055 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.316s user 0.000s sys 0.002s
I20260504 20:01:21.443320 15041 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:21.427103 14969 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:21.419077 14807 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:21.426819 15008 consensus_queue.cc:237] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.458112 15069 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:21.462814 15009 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:21.494951 14807 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:21.499544 14807 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:21.443454 14900 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.442554 15049 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260504 20:01:21.427733 14872 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:21.425726 12863 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.454707 12825 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:21.504875 12825 raft_consensus.cc:606] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.469208 12750 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:21.423574 15003 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260504 20:01:21.482087 15048 raft_consensus.cc:3055] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:21.482349 12741 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:21.482870 15083 raft_consensus.cc:3055] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:21.483920 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:21.486297 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:21.519555 12791 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.453047 15043 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:21.476887 12647 catalog_manager.cc:5671] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 2 to 5, leader changed from ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 5 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:21.494906 12817 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
W20260504 20:01:21.522156 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 96516us
I20260504 20:01:21.466212 14887 consensus_queue.cc:237] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.490109 14726 raft_consensus.cc:2804] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:21.510460 14865 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:21.514812 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1687) took 1821 ms. Trace:
I20260504 20:01:21.478171 15025 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:21.529927 15073 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:21.522451 15049 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:21.525137 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:21.529666 12824 rpcz_store.cc:276] 0504 20:01:19.693215 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:19.693342 (+ 127us) service_pool.cc:224] Handling call
0504 20:01:19.712421 (+ 19079us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:20.562338 (+849917us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:20.562385 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:20.563070 (+ 685us) log.cc:844] Serialized 26 byte log entry
0504 20:01:20.760221 (+197151us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:20.760273 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:20.760368 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:21.514462 (+754094us) raft_consensus.cc:1685] finished
0504 20:01:21.514500 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:21.514702 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":720721,"raft.queue_time_us":721678,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":53,"thread_start_us":311131,"threads_started":2,"wal-append.queue_time_us":246988}
I20260504 20:01:21.532454 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.535326 15049 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:21.529230 14726 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:21.536401 14726 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:21.525602 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 162485us
I20260504 20:01:21.518543 15048 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:21.542636 12791 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:21.503854 15069 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:21.546077 15073 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:21.548013 15069 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:21.549270 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:21.550693 12791 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.551823 12791 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:21.543042 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 81325.2us
I20260504 20:01:21.519788 15083 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:21.543949 12863 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.568742 15069 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:21.529031 14768 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:21.588279 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:21.528604 14891 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 20:01:21.590425 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 49622.6us
I20260504 20:01:21.571187 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:21.603575 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 51793.5us
I20260504 20:01:21.604990 12863 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.605962 12791 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.550572 15049 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:21.614614 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.586998 15073 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260504 20:01:21.602600 15037 consensus_meta.cc:287] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.879s user 0.016s sys 0.001s
I20260504 20:01:21.623710 15049 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:01:21.625322 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 74624us
I20260504 20:01:21.575730 15069 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:21.598165 14865 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:21.625377 12791 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.601258 15054 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260504 20:01:21.617156 15050 consensus_meta.cc:287] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.832s user 0.014s sys 0.000s
I20260504 20:01:21.628345 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:21.618993 14979 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:21.643342 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:21.643640 15085 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:21.644361 15085 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.644737 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:21.650281 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:21.628423 15049 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:21.675807 12791 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:21.683745 15054 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:21.630755 15038 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:21.631731 14865 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:21.640803 15073 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260504 20:01:21.649444 15034 consensus_meta.cc:287] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.889s user 0.014s sys 0.003s
W20260504 20:01:21.689450 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 46443.9us
I20260504 20:01:21.635053 14716 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:21.694665 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 89821.5us
I20260504 20:01:21.626397 15037 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.708169 12749 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:21.713052 12748 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
W20260504 20:01:21.706494 12633 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 0e8eba4dbb8d436f81062ca3bbf46788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 5ff0ef72ec3b4667893a0420d3c7f923 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:21.688124 15049 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:21.717321 12865 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.627521 15048 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.723687 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:21.650998 15085 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:21.724632 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 142607us
I20260504 20:01:21.721401 12750 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:21.640946 15050 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.728672 12714 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:21.695452 14865 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:21.738718 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:21.746309 12892 raft_consensus.cc:2393] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:21.732316 12743 raft_consensus.cc:2393] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:21.740115 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:21.740640 12714 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.750142 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:21.742255 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.746357 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.691349 14979 consensus_queue.cc:1059] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260504 20:01:21.711925 15041 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:21.763899 12894 raft_consensus.cc:2393] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:21.764279 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.716724 14900 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:21.765035 12821 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:21.763903 12817 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:21.766736 12824 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:21.772464 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:21.774658 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:21.775106 12743 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:21.766793 15054 raft_consensus.cc:697] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:21.744896 15073 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:21.773120 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:21.784825 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:21.787688 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 70472.5us
I20260504 20:01:21.793457 12818 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:21.730558 15038 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
W20260504 20:01:21.797461 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 90903.4us
I20260504 20:01:21.768796 15003 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:21.651234 15083 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.804062 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.828720 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:21.791754 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:21.839632 12743 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:21.835407 12714 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:21.779915 15054 consensus_queue.cc:237] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:21.836804 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 56630.7us
I20260504 20:01:21.842658 13497 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.855914 12749 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:21.831902 14716 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:21.847014 15083 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:21.862669 15035 consensus_meta.cc:287] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.939s user 0.017s sys 0.000s
I20260504 20:01:21.887709 13497 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:21.877769 14716 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:21.795557 15073 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:21.883363 14993 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:21.915305 14993 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:21.915782 14993 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:21.903633 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 68341us
I20260504 20:01:21.909317 15077 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260504 20:01:21.886454 15035 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:21.902513 15069 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:21.908804 14716 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:21.917723 15045 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:21.953315 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 119959us
I20260504 20:01:21.922870 15037 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:21.956809 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:22.062297 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:22.092429 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 253303us
I20260504 20:01:21.982553 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:22.189945 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:21.963436 15097 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:22.212503 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:21.962714 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:22.194240 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:21.967654 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 72259.3us
I20260504 20:01:21.956938 15050 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:22.141932 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 311411us
I20260504 20:01:22.209753 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:22.216375 12824 raft_consensus.cc:2393] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:22.212769 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 294025us
I20260504 20:01:21.993937 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:22.245009 12821 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 4
I20260504 20:01:22.244494 12818 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:22.245704 12822 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:22.030958 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 108745us
I20260504 20:01:22.251379 12791 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:21.982852 15077 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:21.963595 15091 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
W20260504 20:01:21.999038 15088 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.478s user 0.002s sys 0.000s
I20260504 20:01:22.254271 15092 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:21.965912 15035 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:22.172210 15080 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:22.279778 15112 consensus_queue.cc:1059] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260504 20:01:22.280732 15080 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.281906 15077 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:22.256587 15073 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:22.288265 15049 consensus_meta.cc:287] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.543s user 0.012s sys 0.005s
I20260504 20:01:22.289007 15049 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.292904 15049 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:21.992430 15101 log.cc:927] Time spent T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260504 20:01:21.992992 15098 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:22.318702 15098 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:22.319556 15098 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:22.315527 14998 consensus_meta.cc:287] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.200s user 0.017s sys 0.000s
I20260504 20:01:21.983129 15045 raft_consensus.cc:697] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:21.992882 15102 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:22.314736 15007 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:21.992412 15108 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:22.216301 15097 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:22.325302 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 75578.5us
I20260504 20:01:22.229642 14942 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:22.037211 15003 raft_consensus.cc:697] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:22.345600 15035 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:22.358945 15006 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:22.359393 15035 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:22.359943 15006 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:22.323690 14726 consensus_meta.cc:287] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.787s user 0.006s sys 0.011s
I20260504 20:01:22.289471 14716 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.354749 15102 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:22.339294 14998 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.368245 14942 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.668109 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1749) took 1037 ms. Trace:
I20260504 20:01:22.374993 15034 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:22.680975 12823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53094 (request call id 523) took 1283 ms. Trace:
W20260504 20:01:22.666623 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 311433us
I20260504 20:01:22.681560 12823 rpcz_store.cc:276] 0504 20:01:21.385307 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:21.386078 (+ 771us) service_pool.cc:224] Handling call
0504 20:01:22.564175 (+1178097us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b54000e3dd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c918470 00007f015c9b731a 00007f016198611e 00007f015af02437 00007f015af023c2 00007f015af022f1 00007f015af0226c 00007f015af01522 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:22.668182 (+104007us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":32304896}
I20260504 20:01:22.672564 12892 rpcz_store.cc:276] 0504 20:01:21.630702 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:21.630871 (+ 169us) service_pool.cc:224] Handling call
0504 20:01:22.637725 (+1006854us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400207bd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:22.668030 (+ 30305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234119936}
I20260504 20:01:22.375011 15050 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:22.548949 12821 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:22.378623 15008 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:22.704118 12863 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:22.499719 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:22.722862 12865 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:22.685006 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:22.739385 12819 raft_consensus.cc:2393] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:22.534533 14807 consensus_meta.cc:287] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.029s user 0.009s sys 0.014s
W20260504 20:01:22.758366 12822 consensus_meta.cc:287] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.511s user 0.007s sys 0.005s
W20260504 20:01:22.528075 15059 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.124s user 0.000s sys 0.007s
W20260504 20:01:22.589399 14865 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.831s user 0.009s sys 0.006s
I20260504 20:01:22.639864 15035 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:22.408573 15009 consensus_meta.cc:287] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.909s user 0.009s sys 0.006s
W20260504 20:01:22.566645 15057 log.cc:927] Time spent T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 1.503s user 0.000s sys 0.004s
I20260504 20:01:22.385416 14823 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:22.584125 15074 log.cc:927] Time spent T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 1.113s user 0.000s sys 0.003s
I20260504 20:01:22.395381 15085 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:22.758947 12822 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
W20260504 20:01:22.759516 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 55500.7us
I20260504 20:01:22.655632 14995 raft_consensus.cc:3055] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:22.755120 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:22.656747 14952 raft_consensus.cc:3055] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:22.559495 14994 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:22.765959 15106 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:22.388659 15082 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:22.567138 15032 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:22.381351 14726 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:22.859109 14993 consensus_meta.cc:287] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.943s user 0.011s sys 0.006s
I20260504 20:01:22.745647 15080 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:22.817986 15048 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:22.959343 13499 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.013s sys 0.030s
W20260504 20:01:22.959875 13499 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.013s sys 0.030s
I20260504 20:01:22.805135 14872 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:22.814200 15077 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:22.744313 15034 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.854319 14995 raft_consensus.cc:740] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:22.854652 15054 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.013s sys 0.023s
I20260504 20:01:22.660497 15003 consensus_queue.cc:237] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.830647 15025 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:22.687127 14966 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:22.755549 15008 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.660925 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:22.825996 15097 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:22.852149 15009 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:22.771862 14807 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.938979 15090 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:22.406402 15045 consensus_queue.cc:237] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.969054 15006 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:22.946695 15049 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:22.952888 15032 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:22.862457 12816 raft_consensus.cc:2393] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:22.789351 14865 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.947872 14952 raft_consensus.cc:740] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:22.856854 14823 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:22.991789 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49576 (request call id 30) took 1534 ms. Trace:
W20260504 20:01:23.003595 15083 consensus_meta.cc:287] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.118s user 0.017s sys 0.004s
W20260504 20:01:22.979322 15054 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s user 0.013s sys 0.023s
I20260504 20:01:22.966576 15106 raft_consensus.cc:697] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:22.766860 15050 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:22.995949 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 234383us
W20260504 20:01:23.003597 12715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.033s sys 0.053s
I20260504 20:01:22.986271 15109 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:22.941936 15085 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.008878 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1545) took 2261 ms. Trace:
I20260504 20:01:22.988566 14716 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:22.991822 12820 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.024s sys 0.073s
W20260504 20:01:23.038300 12820 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s user 0.024s sys 0.073s
I20260504 20:01:23.020606 12825 rpcz_store.cc:276] 0504 20:01:21.450605 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:21.454500 (+ 3895us) service_pool.cc:224] Handling call
0504 20:01:22.984439 (+1529939us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b54003163a0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c92b900 00007f015c9a8a32 00007f0161984cca 00007f015aefd934 00007f015aefd8c2 00007f015aefd7f1 00007f015aefd76c 00007f015aefca22 00007f015ace8bcc 00007f015ace8006
0504 20:01:22.984709 (+ 270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26984192}
I20260504 20:01:22.966877 14993 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:23.030428 12715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s user 0.034s sys 0.053s
I20260504 20:01:22.978123 15077 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:23.035853 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:22.955868 15082 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.016885 15090 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.034199 12747 rpcz_store.cc:276] 0504 20:01:20.742565 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:20.745565 (+ 3000us) service_pool.cc:224] Handling call
0504 20:01:20.746014 (+ 449us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:20.759649 (+ 13635us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:20.759697 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:20.779485 (+ 19788us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b2c00056c38. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015b2a8c81 00007f015b2b85e2 00007f015c9e9de1 00007f015c9ae81a 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22
0504 20:01:20.779954 (+ 469us) log.cc:844] Serialized 26 byte log entry
0504 20:01:21.362084 (+582130us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:21.362131 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:21.362198 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:22.583062 (+1220864us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:23.002253 (+419191us) raft_consensus.cc:1685] finished
0504 20:01:23.002284 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:23.003479 (+ 1195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33658,"raft.queue_time_us":16791,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":16040,"spinlock_wait_cycles":8967424,"thread_start_us":581717,"threads_started":1,"wal-append.queue_time_us":618091}
W20260504 20:01:23.036605 13530 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.502s user 0.014s sys 0.050s
I20260504 20:01:22.990934 14966 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:23.058430 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:23.060853 14229 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:23.067118 14952 consensus_queue.cc:260] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.076304 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1753) took 2411 ms. Trace:
I20260504 20:01:23.108093 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:23.109556 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:23.110214 12893 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:23.029202 13529 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 539) took 1575 ms (client timeout 1721 ms). Trace:
I20260504 20:01:23.103724 12820 rpcz_store.cc:276] 0504 20:01:20.635516 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:20.681962 (+ 46446us) service_pool.cc:224] Handling call
0504 20:01:20.682231 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:20.682423 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 4
0504 20:01:22.099935 (+1417512us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b18004343e0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f015c55585c 00007f01598f3c0b 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f015990da95 00007f01603b1837 00007f015c9b8d87 00007f015c9c849f
0504 20:01:22.100164 (+ 229us) log.cc:844] Serialized 14 byte log entry
0504 20:01:23.045647 (+945483us) log.cc:844] Serialized 14 byte log entry
0504 20:01:23.045941 (+ 294us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:23.046012 (+ 71us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:23.046063 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:23.046159 (+ 96us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:23.046320 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7673,"prepare.queue_time_us":321230,"prepare.run_cpu_time_us":4198,"prepare.run_wall_time_us":26903,"raft.queue_time_us":639,"raft.run_wall_time_us":15974,"spinlock_wait_cycles":66053760,"thread_start_us":1190961,"threads_started":2,"wal-append.queue_time_us":929222}
I20260504 20:01:23.100809 15114 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:23.116750 15114 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.117682 15050 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:23.041482 15081 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:23.118451 15050 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:23.118661 15081 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.115535 12863 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:23.108943 12898 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:23.043231 14998 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:23.078727 13530 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.544s user 0.014s sys 0.050s
I20260504 20:01:23.135742 12899 raft_consensus.cc:1240] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:23.128902 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:23.101002 15136 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:23.119052 15050 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.134569 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:23.099061 15097 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:23.136960 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:23.062907 15083 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:23.137578 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:23.030325 15049 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.139015 12716 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:23.144554 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:23.146227 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:23.070659 14995 consensus_queue.cc:260] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.044152 15040 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:23.072350 15124 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:23.152299 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:23.168816 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:23.151990 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:23.110877 14899 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:23.137523 12747 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:23.158144 12891 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:23.159026 12895 raft_consensus.cc:2368] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:23.111747 15037 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:23.121098 14716 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:23.057523 15077 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:23.159322 13497 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.066s sys 0.123s
I20260504 20:01:23.169011 12743 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:23.155108 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:23.183753 13497 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.806s user 0.066s sys 0.123s
W20260504 20:01:23.185369 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 843801us
I20260504 20:01:23.186268 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:23.125821 14942 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:23.172128 12748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1550) took 2419 ms (client timeout 1744 ms). Trace:
W20260504 20:01:23.187498 12748 rpcz_store.cc:269] 0504 20:01:20.744073 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:20.760192 (+ 16119us) service_pool.cc:224] Handling call
0504 20:01:23.163105 (+2402913us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b54000dcad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:23.163258 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2155896704}
I20260504 20:01:23.189265 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:23.189046 14716 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.190126 12748 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:23.192785 14716 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:23.195135 15129 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260504 20:01:23.180236 12741 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260504 20:01:23.181190 12894 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1623) took 2418 ms (client timeout 2767 ms). Trace:
W20260504 20:01:23.199100 12894 rpcz_store.cc:269] 0504 20:01:20.754879 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:20.755068 (+ 189us) service_pool.cc:224] Handling call
0504 20:01:23.172901 (+2417833us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003ba454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:23.173081 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2317504768}
W20260504 20:01:23.120739 13529 rpcz_store.cc:269] 0504 20:01:21.454032 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:22.660402 (+1206370us) service_pool.cc:224] Handling call
0504 20:01:23.024945 (+364543us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:23.202108 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:23.203591 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:23.203644 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:23.204680 12894 raft_consensus.cc:2368] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
W20260504 20:01:23.204944 12714 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.882s user 0.028s sys 0.014s
W20260504 20:01:23.205504 12714 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.883s user 0.028s sys 0.014s
W20260504 20:01:23.205996 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 960538us
I20260504 20:01:23.206470 12741 raft_consensus.cc:1275] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260504 20:01:23.209386 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:23.042963 15109 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.214236 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:23.171339 14998 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:23.220407 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:23.224941 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:23.198891 14995 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:23.157459 15083 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.197085 15114 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:23.226871 12894 raft_consensus.cc:1275] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:23.172132 15107 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:23.228477 13497 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:23.187345 15009 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:23.206933 12890 raft_consensus.cc:1240] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:23.212819 14872 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:23.207566 15034 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:23.243047 12743 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:23.254845 12748 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:23.200299 15037 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:23.260912 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 107607us
W20260504 20:01:23.265523 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 124215us
I20260504 20:01:23.183688 14899 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.230046 14990 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:23.229532 14823 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:23.271605 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:23.152904 15136 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:23.309978 12788 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.010s sys 0.003s
I20260504 20:01:23.313328 12743 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
W20260504 20:01:23.321354 12788 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s user 0.010s sys 0.003s
W20260504 20:01:23.321904 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:23.316728 12747 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:23.318780 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:23.323112 12820 raft_consensus.cc:1813] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Handling vote request from an unknown peer 732c3560a688494196d22a1a48f89968
I20260504 20:01:23.323779 12820 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:23.232935 15102 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:23.320765 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:23.255427 14807 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:23.326681 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:23.328708 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:23.326835 12897 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:23.331336 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:23.332296 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:23.332242 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:23.266254 15008 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:23.257825 14726 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:23.177655 15106 consensus_queue.cc:237] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:23.172850 15040 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:23.268319 14865 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:23.333897 12821 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.002s sys 0.002s
W20260504 20:01:23.374538 12821 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.606s user 0.003s sys 0.002s
I20260504 20:01:23.336374 12825 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:23.343735 12891 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:23.339890 12819 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:23.340196 12820 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:23.199271 15124 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.336968 12822 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
W20260504 20:01:23.361042 12791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.006s sys 0.017s
I20260504 20:01:23.342231 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:23.337538 12823 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:23.420645 12816 raft_consensus.cc:2393] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:23.259851 15009 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.356971 15138 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:23.267140 14872 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.422029 15138 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:23.415544 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:23.356957 15132 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.315s user 0.001s sys 0.001s
I20260504 20:01:23.349242 14993 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:23.366144 15085 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:23.394142 15102 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:23.441723 14899 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:23.442965 14726 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:23.443149 15077 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:23.445060 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:23.444921 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:23.335969 15117 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:23.419929 12791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s user 0.007s sys 0.017s
I20260504 20:01:23.423875 15009 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:23.458626 15150 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:23.460512 15127 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.265s user 0.000s sys 0.001s
I20260504 20:01:23.438866 14807 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:23.470614 15141 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:23.470711 14872 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:23.475199 15040 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:23.476446 15083 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:23.430094 13526 pending_rounds.cc:85] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Aborting all ops after (but not including) 7
W20260504 20:01:23.583902 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:24.558879 12788 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.236s user 0.007s sys 0.010s
W20260504 20:01:24.572887 12788 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.250s user 0.007s sys 0.010s
W20260504 20:01:24.072425 12716 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.032s sys 0.084s
W20260504 20:01:24.573539 12716 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.245s user 0.038s sys 0.102s
I20260504 20:01:23.797189 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:24.023074 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1547) took 3279 ms. Trace:
W20260504 20:01:24.297089 12818 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.564s user 0.056s sys 0.139s
I20260504 20:01:23.521454 12820 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
W20260504 20:01:24.098493 12893 consensus_meta.cc:287] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.988s user 0.013s sys 0.002s
W20260504 20:01:23.621743 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 80365.5us
W20260504 20:01:24.488869 12821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.094s user 0.002s sys 0.015s
W20260504 20:01:24.470508 12715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.056s user 0.000s sys 0.021s
W20260504 20:01:24.573902 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 1.24524e+06us
W20260504 20:01:24.578027 12715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.163s user 0.000s sys 0.024s
I20260504 20:01:23.543146 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:24.577680 12821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.200s user 0.007s sys 0.015s
I20260504 20:01:24.577086 12893 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:24.559831 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1741) took 1016 ms. Trace:
W20260504 20:01:24.601711 12893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1630) took 1494 ms (client timeout 1852 ms). Trace:
I20260504 20:01:23.434001 15090 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:24.574460 13532 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 354) took 1224 ms. Trace:
W20260504 20:01:24.575173 12818 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.842s user 0.063s sys 0.145s
I20260504 20:01:24.614697 12825 rpcz_store.cc:276] 0504 20:01:23.543562 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.544510 (+ 948us) service_pool.cc:224] Handling call
0504 20:01:23.545045 (+ 535us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:23.545305 (+ 260us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:23.545348 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:23.545409 (+ 61us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:23.545468 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:23.545547 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:24.559737 (+1014190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:24.615861 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1721) took 2360 ms. Trace:
I20260504 20:01:24.562135 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:24.588117 13528 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 547) took 1402 ms (client timeout 1757 ms). Trace:
I20260504 20:01:24.617312 12816 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:24.594791 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 1.22304e+06us
I20260504 20:01:24.600574 12899 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:24.618076 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1738) took 1076 ms. Trace:
I20260504 20:01:23.476963 15116 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:24.579588 12863 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:23.527664 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:23.433873 15082 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:23.417318 15035 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:23.393342 15133 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:24.579113 12823 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:24.606303 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1735) took 1246 ms. Trace:
I20260504 20:01:23.505163 12819 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
W20260504 20:01:24.573160 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 2.32188e+06us
W20260504 20:01:24.628396 12819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1737) took 1297 ms (client timeout 1626 ms). Trace:
I20260504 20:01:24.618673 12816 rpcz_store.cc:276] 0504 20:01:23.542188 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.543735 (+ 1547us) service_pool.cc:224] Handling call
0504 20:01:24.618003 (+1074268us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:24.554085 15025 consensus_meta.cc:287] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.571s user 0.021s sys 0.000s
W20260504 20:01:24.614753 12893 rpcz_store.cc:269] 0504 20:01:23.107382 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.109288 (+ 1906us) service_pool.cc:224] Handling call
0504 20:01:24.579818 (+1470530us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:24.627594 12820 rpcz_store.cc:276] 0504 20:01:23.330582 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.332031 (+ 1449us) service_pool.cc:224] Handling call
0504 20:01:24.576832 (+1244801us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37376}
I20260504 20:01:23.446451 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:24.023304 15128 log.cc:927] Time spent T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.740s user 0.000s sys 0.001s
W20260504 20:01:23.719048 15135 log.cc:927] Time spent T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32: Append to log took a long time: real 0.590s user 0.003s sys 0.000s
W20260504 20:01:23.818985 15079 log.cc:927] Time spent T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 2.186s user 0.006s sys 0.003s
I20260504 20:01:23.686050 15049 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:24.618019 13532 rpcz_store.cc:276] 0504 20:01:23.350531 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.351263 (+ 732us) service_pool.cc:224] Handling call
0504 20:01:23.796996 (+445733us) spinlock_profiling.cc:243] Waited 1.03 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:24.574376 (+777380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2375936}
I20260504 20:01:24.621917 12863 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:24.603004 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1726) took 1938 ms. Trace:
I20260504 20:01:23.683305 15149 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:24.619331 13533 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
W20260504 20:01:24.032265 15137 log.cc:927] Time spent T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.391s user 0.003s sys 0.000s
W20260504 20:01:23.538358 15118 log.cc:927] Time spent T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.379s user 0.002s sys 0.000s
W20260504 20:01:24.678253 12864 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.4:41009 timed out after 1.554s (SENT) [suppressed 5 similar messages]
W20260504 20:01:23.640758 14994 consensus_meta.cc:287] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.676s user 0.013s sys 0.004s
I20260504 20:01:24.625909 13531 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 355) took 1232 ms. Trace:
I20260504 20:01:24.623704 15083 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:24.627719 12823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1739) took 1085 ms. Trace:
I20260504 20:01:24.702962 12823 rpcz_store.cc:276] 0504 20:01:23.542607 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.542780 (+ 173us) service_pool.cc:224] Handling call
0504 20:01:24.627641 (+1084861us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:23.719192 13524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 537) took 2599 ms. Trace:
I20260504 20:01:24.613564 14726 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:24.733973 12713 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.319s user 0.023s sys 0.046s
I20260504 20:01:24.631804 15148 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:23.788966 14979 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:24.614086 14872 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:24.590265 13526 pending_rounds.cc:107] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Aborting uncommitted NO_OP operation due to leader change: 6.8
W20260504 20:01:24.628976 12819 rpcz_store.cc:269] 0504 20:01:23.331276 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.331986 (+ 710us) service_pool.cc:224] Handling call
0504 20:01:24.628306 (+1296320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:24.637895 12716 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:24.767599 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 129857us
I20260504 20:01:23.791205 15092 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:23.518918 15048 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:24.628134 12715 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:24.798612 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 170570us
I20260504 20:01:24.616376 12818 rpcz_store.cc:276] 0504 20:01:22.255147 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:22.255461 (+ 314us) service_pool.cc:224] Handling call
0504 20:01:22.278779 (+ 23318us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:22.279086 (+ 307us) raft_consensus.cc:1534] Early marking committed up to index 3
0504 20:01:22.382064 (+102978us) log.cc:844] Serialized 14 byte log entry
0504 20:01:24.615333 (+2233269us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:24.615427 (+ 94us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:24.615497 (+ 70us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:24.615595 (+ 98us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:24.615784 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":429730,"prepare.queue_time_us":669667,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":72,"raft.queue_time_us":422728,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":13101,"thread_start_us":1984855,"threads_started":2}
W20260504 20:01:24.037181 15098 consensus_meta.cc:287] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.717s user 0.018s sys 0.002s
I20260504 20:01:24.031425 15136 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:24.035735 15078 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:23.477931 14966 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:23.567868 15018 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:23.772670 15037 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:23.527967 15109 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:23.644699 15140 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:23.776708 15124 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:23.777827 14823 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:24.582221 15143 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:24.617266 13528 rpcz_store.cc:269] 0504 20:01:23.185203 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.187522 (+ 2319us) service_pool.cc:224] Handling call
0504 20:01:23.444819 (+257297us) spinlock_profiling.cc:243] Waited 6.09 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:24.587990 (+1143171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14018432}
I20260504 20:01:24.574784 12744 rpcz_store.cc:276] 0504 20:01:20.743154 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:20.746475 (+ 3321us) service_pool.cc:224] Handling call
0504 20:01:20.746910 (+ 435us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:20.768134 (+ 21224us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:20.768168 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:20.777274 (+ 9106us) log.cc:844] Serialized 26 byte log entry
0504 20:01:21.999419 (+1222145us) spinlock_profiling.cc:243] Waited 9.6 ms on lock 0x7b18003e2a80. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d471 00007f015c969def
0504 20:01:22.314275 (+314856us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:22.314341 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:22.314442 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:24.022108 (+1707666us) raft_consensus.cc:1685] finished
0504 20:01:24.022144 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:24.022968 (+ 824us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48756,"raft.queue_time_us":26493,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"spinlock_wait_cycles":22070016,"thread_start_us":314589,"threads_started":1,"wal-append.queue_time_us":2184364}
I20260504 20:01:23.537427 15081 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:24.258105 15151 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:24.945571 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1553) took 4179 ms. Trace:
I20260504 20:01:23.456984 15050 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:24.658725 12821 rpcz_store.cc:276] 0504 20:01:22.664641 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:22.730396 (+ 65755us) service_pool.cc:224] Handling call
0504 20:01:22.730834 (+ 438us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:22.731139 (+ 305us) raft_consensus.cc:1534] Early marking committed up to index 4
0504 20:01:23.376460 (+645321us) log.cc:844] Serialized 14 byte log entry
0504 20:01:24.602665 (+1226205us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:24.602730 (+ 65us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:24.602778 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:24.602841 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:24.602948 (+ 107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2302,"prepare.queue_time_us":239655,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":8572,"thread_start_us":1809625,"threads_started":2}
W20260504 20:01:24.924034 15045 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.323s user 0.023s sys 0.047s
I20260504 20:01:24.607725 14899 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:24.694739 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 72931.8us
W20260504 20:01:23.595296 15131 log.cc:927] Time spent T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.371s user 0.000s sys 0.002s
I20260504 20:01:24.791957 14979 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:24.696127 13531 rpcz_store.cc:276] 0504 20:01:23.395061 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.443760 (+ 48699us) service_pool.cc:224] Handling call
0504 20:01:24.625695 (+1181935us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:24.620736 15141 raft_consensus.cc:697] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:24.643818 15090 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:24.639079 15025 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:24.705368 13499 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.435s user 0.017s sys 0.030s
W20260504 20:01:24.978607 13499 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.708s user 0.017s sys 0.031s
I20260504 20:01:24.709704 13524 rpcz_store.cc:276] 0504 20:01:21.119594 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:21.122500 (+ 2906us) service_pool.cc:224] Handling call
0504 20:01:21.135256 (+ 12756us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:21.272427 (+137171us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:23.718722 (+2446295us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b540051d1d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:23.719060 (+ 338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110469888}
I20260504 20:01:24.710736 15083 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:24.602075 14807 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:24.654436 13529 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
W20260504 20:01:24.711726 15097 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.477s user 0.008s sys 0.041s
W20260504 20:01:24.994875 15097 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.762s user 0.008s sys 0.042s
I20260504 20:01:24.995735 14807 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:24.701324 15041 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:24.996585 13529 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 353) took 1652 ms (client timeout 2124 ms). Trace:
I20260504 20:01:24.996836 15041 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:24.997071 13529 rpcz_store.cc:269] 0504 20:01:23.344561 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.347073 (+ 2512us) service_pool.cc:224] Handling call
0504 20:01:24.996504 (+1649431us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:24.621315 15102 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.000346 15102 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:25.000991 15102 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:24.700378 15049 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:25.002223 15049 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.004300 15049 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:25.004839 15049 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:24.776693 12716 leader_election.cc:400] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:01:25.005965 12716 leader_election.cc:403] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260504 20:01:25.006474 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 237929us
I20260504 20:01:25.007315 12716 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:24.805182 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:24.812945 12865 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.178s user 0.011s sys 0.012s
W20260504 20:01:25.009430 12865 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.375s user 0.012s sys 0.012s
W20260504 20:01:25.009855 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 1.57324e+06us
I20260504 20:01:25.012223 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:24.636461 15035 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.013198 12894 raft_consensus.cc:1240] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:24.930210 15148 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.013212 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:25.014322 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:25.015061 12893 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:24.608152 15117 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:24.871619 12714 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.491s user 0.032s sys 0.069s
W20260504 20:01:25.015969 12714 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.635s user 0.032s sys 0.070s
I20260504 20:01:24.791656 15092 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:24.891333 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1719) took 2673 ms. Trace:
I20260504 20:01:24.892064 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:24.624717 15116 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:24.766636 13526 raft_consensus.cc:2889] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:25.005288 15049 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:24.635623 15150 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:24.939880 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 163677us
I20260504 20:01:24.704016 15009 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:24.736636 14994 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:24.862391 15054 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:24.685356 13533 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 356) took 1159 ms. Trace:
I20260504 20:01:24.946959 12742 rpcz_store.cc:276] 0504 20:01:20.766092 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:20.766742 (+ 650us) service_pool.cc:224] Handling call
0504 20:01:20.767051 (+ 309us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:21.342703 (+575652us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:24.945222 (+3602519us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b5400027e54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:24.945472 (+ 250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1222512384}
I20260504 20:01:24.619619 15077 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:24.886829 15133 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:24.938513 14995 consensus_meta.cc:287] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.702s user 0.016s sys 0.007s
W20260504 20:01:25.175206 15156 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.249s user 0.000s sys 0.000s
I20260504 20:01:24.975775 14899 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:25.176163 13496 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.229s user 0.045s sys 0.065s
I20260504 20:01:24.645016 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 282) took 3239 ms. Trace:
W20260504 20:01:25.176671 13496 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.522s user 0.045s sys 0.066s
I20260504 20:01:24.954360 15152 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.200s
I20260504 20:01:25.001668 15102 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:24.969184 12863 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:24.792111 14872 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:24.737999 15149 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:24.951821 15045 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.354s user 0.023s sys 0.047s
W20260504 20:01:24.974308 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 345116us
W20260504 20:01:24.978964 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 1.70849e+06us
W20260504 20:01:24.734550 12713 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.320s user 0.024s sys 0.046s
I20260504 20:01:24.866631 15098 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:24.949980 15158 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:24.982527 15163 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:24.967679 15143 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:24.964630 15140 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:24.968405 15124 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:25.020210 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:25.023133 12817 rpcz_store.cc:276] 0504 20:01:22.218188 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:22.218369 (+ 181us) service_pool.cc:224] Handling call
0504 20:01:22.218861 (+ 492us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:22.219189 (+ 328us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:22.219227 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:22.219984 (+ 757us) log.cc:844] Serialized 26 byte log entry
0504 20:01:22.386142 (+166158us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:22.386203 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:22.386314 (+ 111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:23.779393 (+1393079us) spinlock_profiling.cc:243] Waited 2.64 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:24.866332 (+1086939us) raft_consensus.cc:1685] finished
0504 20:01:24.866361 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:24.891213 (+ 24852us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6060928,"thread_start_us":165582,"threads_started":1,"wal-append.queue_time_us":759467}
I20260504 20:01:24.964578 15081 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:25.026934 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:24.948203 14823 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:24.875461 15048 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.017326 15117 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:24.954178 15161 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:24.926142 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 361) took 1865 ms. Trace:
W20260504 20:01:25.147729 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 123371us
I20260504 20:01:24.977576 15090 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:25.174254 15054 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.175522 15077 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:24.933022 15136 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:24.964430 15151 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:24.942143 15078 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.015506 15148 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:25.520561 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 113261us
I20260504 20:01:25.183496 12746 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:25.169178 13533 rpcz_store.cc:276] 0504 20:01:23.526238 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.526718 (+ 480us) service_pool.cc:224] Handling call
0504 20:01:24.685284 (+1158566us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:25.534487 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:25.543628 12745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1546) took 4637 ms (client timeout 2032 ms). Trace:
W20260504 20:01:25.546262 13497 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.026s sys 0.038s
I20260504 20:01:25.064997 14807 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:25.546707 13497 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.507s user 0.026s sys 0.038s
W20260504 20:01:25.544106 12745 rpcz_store.cc:269] 0504 20:01:20.742868 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:20.746373 (+ 3505us) service_pool.cc:224] Handling call
0504 20:01:25.343422 (+4597049us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b5400011654. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:25.379916 (+ 36494us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2227938048}
I20260504 20:01:25.548472 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:25.547197 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:25.553115 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 513464us
I20260504 20:01:25.555585 15150 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:25.531273 15032 consensus_meta.cc:287] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 2.501s user 0.013s sys 0.008s
I20260504 20:01:25.557250 12790 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:25.551690 12894 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:01:25.560006 13499 connection.cc:254] RPC call timeout handler was delayed by -2.0514122490000091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
I20260504 20:01:25.561580 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:25.165954 15009 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:25.152987 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 352) took 3622 ms. Trace:
W20260504 20:01:25.553409 12791 connection.cc:254] RPC call timeout handler was delayed by -3.5666185769999856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
I20260504 20:01:25.545365 15090 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:25.555637 13497 leader_election.cc:336] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.542s (SENT)
W20260504 20:01:25.569114 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 600066us
I20260504 20:01:25.636255 13497 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:25.564546 12892 rpcz_store.cc:276] 0504 20:01:23.021330 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.021532 (+ 202us) service_pool.cc:224] Handling call
0504 20:01:23.040884 (+ 19352us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:23.109591 (+ 68707us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:23.109627 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:23.109670 (+ 43us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:23.109704 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:24.886330 (+1776626us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b54002cc4d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:24.886393 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:24.886575 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73474816}
I20260504 20:01:25.209946 12824 raft_consensus.cc:1275] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:25.585171 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:25.637907 12819 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:25.526816 15083 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:25.375322 15138 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:25.179404 14979 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:25.534065 13527 rpcz_store.cc:276] 0504 20:01:21.405408 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:21.407966 (+ 2558us) service_pool.cc:224] Handling call
0504 20:01:21.447571 (+ 39605us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:21.547210 (+ 99639us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:21.547254 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:21.547314 (+ 60us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:21.547385 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:24.612444 (+3065059us) spinlock_profiling.cc:243] Waited 99.3 ms on lock 0x7b540058c354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:24.620593 (+ 8149us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:24.638830 (+ 18237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228482432}
I20260504 20:01:25.589475 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:25.589890 15054 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:25.590108 12820 raft_consensus.cc:1240] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:25.677304 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:25.678702 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:25.681062 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:25.102337 15157 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:01:25.589478 12818 raft_consensus.cc:1240] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:25.590696 15077 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:25.591068 14966 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:25.591846 12825 raft_consensus.cc:1275] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:25.687330 12820 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:25.552002 15124 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.594547 14823 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:25.597368 14899 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:25.563575 15150 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:25.546903 15081 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:25.634305 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 82679.2us
I20260504 20:01:25.543391 14726 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:25.676437 14768 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:25.646231 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 90626.3us
I20260504 20:01:25.647471 12896 raft_consensus.cc:2393] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:01:25.601046 15168 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260504 20:01:25.651086 12897 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:25.698390 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:25.584694 15001 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:25.686163 14995 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:25.669592 15169 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:25.674955 15035 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:25.675519 15045 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:25.681571 12819 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:25.590237 15049 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:25.694875 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:25.696983 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:25.698390 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:25.634249 15097 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:25.720773 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:25.734709 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:25.734889 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:25.801301 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:25.657686 15017 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:25.787060 15082 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:25.739857 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:25.543349 15073 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:25.745903 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:25.558911 15163 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.547302 15158 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.687587 14887 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:25.689273 15138 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:25.754428 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:25.639889 15155 consensus_queue.cc:1048] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.158s
I20260504 20:01:25.757876 15149 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:25.729292 15167 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:25.741518 15150 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:25.765126 15041 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:25.767954 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:25.712736 14979 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:25.787029 14768 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:25.779524 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 215340us
I20260504 20:01:25.781785 12820 raft_consensus.cc:2368] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:01:25.833947 12714 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:25.642216 15009 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:25.565269 14807 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.722147 14966 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:25.305650 15141 consensus_queue.cc:237] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.800222 12825 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:25.801455 12816 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:25.801466 12743 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:25.570327 15032 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.802804 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:25.802067 12819 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:25.768419 13525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 284) took 2686 ms. Trace:
I20260504 20:01:25.811275 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:25.815678 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 178794us
I20260504 20:01:26.022976 12863 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:25.820847 12791 connection.cc:254] RPC call timeout handler was delayed by -2.62431857699994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2767s for a response
I20260504 20:01:25.824548 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:25.653308 15136 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.831969 12823 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
W20260504 20:01:26.023654 12791 connection.cc:254] RPC call timeout handler was delayed by -2.325185016999967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
W20260504 20:01:26.024084 12791 connection.cc:254] RPC call timeout handler was delayed by -2.1892850169999747s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18520000000000003s for a response
W20260504 20:01:26.024307 12791 connection.cc:254] RPC call timeout handler was delayed by -2.1163850169999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19330000000000003s for a response
W20260504 20:01:26.024595 12791 connection.cc:254] RPC call timeout handler was delayed by -2.1163850169999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19330000000000003s for a response
W20260504 20:01:26.024756 12791 connection.cc:254] RPC call timeout handler was delayed by -2.1019850169999472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
I20260504 20:01:26.025388 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.026058 15078 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:25.832865 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:26.026893 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:25.797098 14995 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.805234 15097 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.814658 15073 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.802052 15045 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:26.004236 14966 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:26.012583 12865 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:26.032815 12791 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:26.033136 12896 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1754) took 4308 ms (client timeout 1616 ms). Trace:
I20260504 20:01:25.854764 12821 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:25.977707 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:26.012627 12898 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:26.034472 12821 raft_consensus.cc:606] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:26.035015 12821 raft_consensus.cc:614] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260504 20:01:26.016551 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 173739us
I20260504 20:01:26.004060 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:25.823606 15077 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:26.036563 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:25.831405 15092 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.679s user 0.009s sys 0.003s
I20260504 20:01:25.802959 15169 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.810595 15082 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:25.181704 14994 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:26.027597 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:25.966323 15173 log.cc:927] Time spent T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.215s user 0.000s sys 0.001s
I20260504 20:01:26.028697 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:26.229630 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:26.012315 15043 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:25.833057 15150 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:26.033607 12896 rpcz_store.cc:269] 0504 20:01:21.710019 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:21.710210 (+ 191us) service_pool.cc:224] Handling call
0504 20:01:22.407895 (+697685us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:25.971243 (+3563348us) spinlock_profiling.cc:243] Waited 2.91 s on lock 0x7b5400011154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:26.018092 (+ 46849us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6696583168}
I20260504 20:01:26.034971 12742 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:25.846005 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:26.039163 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:25.820327 15049 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:25.845963 14726 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:26.041569 12863 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:25.821648 15138 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.987123 15102 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:25.977075 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:25.817294 14887 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:25.857856 15048 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:26.111822 12821 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:26.182991 15092 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:26.184581 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:26.364897 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:26.011740 15006 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:26.012048 15117 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:26.230016 12743 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:25.834342 15172 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:26.018651 14891 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:26.021857 15106 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:26.374817 14891 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:26.359396 14768 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:26.376972 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.377449 14891 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:26.378152 14891 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:26.378561 14891 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:26.378904 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.379854 13527 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:25.836787 14974 consensus_queue.cc:1059] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.215s
I20260504 20:01:25.847332 15041 consensus_queue.cc:1048] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.386s
I20260504 20:01:26.038959 15181 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:26.027638 15163 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:26.183326 15152 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:26.032902 15098 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:25.841233 14979 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:26.399160 14979 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:26.290329 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:26.399830 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.401209 13533 raft_consensus.cc:2410] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260504 20:01:26.401005 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:26.402474 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:26.286703 12897 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:26.258420 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.232100 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.407140 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.407287 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.408241 13532 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:26.316128 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:26.408578 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.409935 15041 raft_consensus.cc:3055] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
W20260504 20:01:26.312834 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 290129us
I20260504 20:01:26.238767 14899 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:26.260313 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:26.234889 13528 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:26.234539 15151 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:26.259860 15025 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:25.834730 15167 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:26.350315 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 308821us
I20260504 20:01:26.347110 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:26.349491 12819 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:26.185267 15077 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:26.355208 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 342700us
I20260504 20:01:26.262318 15140 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:26.187493 15078 consensus_queue.cc:1059] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.159s
I20260504 20:01:26.294283 15124 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:26.350145 15171 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.512s user 0.001s sys 0.036s
I20260504 20:01:26.183974 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:25.601567 13530 rpcz_store.cc:276] 0504 20:01:21.516609 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:21.518013 (+ 1404us) service_pool.cc:224] Handling call
0504 20:01:21.530873 (+ 12860us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:21.534015 (+ 3142us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:21.534051 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:21.534655 (+ 604us) log.cc:844] Serialized 26 byte log entry
0504 20:01:22.640021 (+1105366us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f01598f382c 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab
0504 20:01:23.157326 (+517305us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:23.160469 (+ 3143us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:23.175058 (+ 14589us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:24.951318 (+1776260us) raft_consensus.cc:1685] finished
0504 20:01:24.963740 (+ 12422us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:25.138551 (+174811us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":125,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":11997952,"thread_start_us":1601042,"threads_started":1,"wal-append.queue_time_us":1531594}
I20260504 20:01:26.254503 15043 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:26.368052 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:26.351697 15081 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:26.346155 14726 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:26.375044 15106 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:26.236717 12818 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:26.355010 15032 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:26.411404 12894 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:26.412182 12820 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:26.414817 12714 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:26.357295 15049 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:26.382157 15181 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:26.362398 15102 consensus_queue.cc:1059] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260504 20:01:26.374742 15172 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:26.416684 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.993651 13531 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 358) took 1015 ms. Trace:
I20260504 20:01:26.994590 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:26.994918 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1787) took 1402 ms. Trace:
I20260504 20:01:26.996441 12822 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:26.384522 15006 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:26.703516 12743 raft_consensus.cc:1240] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:26.422693 12790 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:26.907150 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:27.000594 12821 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:27.000415 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.002177 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:26.591298 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 408055us
I20260504 20:01:27.003926 15077 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.004446 12791 leader_election.cc:400] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:27.004940 12791 leader_election.cc:403] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:26.512480 15054 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:27.006479 12791 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:27.006197 15054 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:26.829623 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:26.797293 15189 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:26.460887 15117 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:26.680711 15180 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:26.021929 13525 rpcz_store.cc:276] 0504 20:01:23.082382 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.120054 (+ 37672us) service_pool.cc:224] Handling call
0504 20:01:23.201615 (+ 81561us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:23.432845 (+231230us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:23.445186 (+ 12341us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:23.465928 (+ 20742us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:23.478694 (+ 12766us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:25.768125 (+2289431us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54003cd054. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:25.768189 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:25.768336 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":457121664}
I20260504 20:01:26.746418 15158 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:26.920194 15163 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:26.419979 14768 consensus_queue.cc:1059] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.615s
I20260504 20:01:26.409596 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:26.997711 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:26.389835 14807 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:26.378923 14891 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:27.001608 12743 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:01:27.013218 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 598457us
I20260504 20:01:27.019236 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1592) took 1225 ms. Trace:
I20260504 20:01:26.994421 13531 rpcz_store.cc:276] 0504 20:01:25.978044 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:26.028789 (+ 50745us) service_pool.cc:224] Handling call
0504 20:01:26.993545 (+964756us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:27.051141 12742 rpcz_store.cc:276] 0504 20:01:25.793694 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:26.346703 (+553009us) service_pool.cc:224] Handling call
0504 20:01:27.019144 (+672441us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:27.013578 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.012535 12863 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:27.055075 12863 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:27.063060 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:27.064097 12745 raft_consensus.cc:2393] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:27.064136 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.066751 12791 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:27.016551 12747 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:26.391695 15152 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:26.995556 12819 rpcz_store.cc:276] 0504 20:01:25.592552 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.800998 (+208446us) service_pool.cc:224] Handling call
0504 20:01:26.994830 (+1193832us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:27.063205 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:26.755569 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:27.068584 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 59906.2us
I20260504 20:01:27.029961 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:26.410362 15041 raft_consensus.cc:740] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:27.078490 12743 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:27.080128 13532 raft_consensus.cc:1813] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Handling vote request from an unknown peer 732c3560a688494196d22a1a48f89968
I20260504 20:01:26.911466 14995 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.010741 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.081039 12714 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:27.133543 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.143695 12743 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:27.143707 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 62828us
I20260504 20:01:27.144667 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.014794 15025 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:27.081770 12864 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.010s sys 0.009s
I20260504 20:01:27.123102 15043 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:27.123741 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.015132 15151 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:27.153192 12745 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:27.029537 15049 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.018292 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.041853 15117 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:27.125270 14966 consensus_meta.cc:287] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.867s user 0.012s sys 0.000s
I20260504 20:01:27.020260 14726 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:27.119354 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:27.032410 15181 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.082515 15097 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.170444 12820 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:27.130151 15169 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:27.141566 15006 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.132567 13532 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:27.056495 15124 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:27.152266 15007 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:27.009738 15054 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.063414 15082 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:27.014359 15032 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:27.150542 12864 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.702s user 0.011s sys 0.009s
I20260504 20:01:27.037557 15073 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:27.193293 12891 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1764) took 4026 ms (client timeout 1757 ms). Trace:
W20260504 20:01:27.193766 12891 rpcz_store.cc:269] 0504 20:01:23.166411 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.180149 (+ 13738us) service_pool.cc:224] Handling call
0504 20:01:27.193056 (+4012907us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54003b9f54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:27.193209 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":306809216}
W20260504 20:01:27.195768 12897 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1767) took 2184 ms (client timeout 2372 ms). Trace:
W20260504 20:01:27.196228 12897 rpcz_store.cc:269] 0504 20:01:25.011779 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.011910 (+ 131us) service_pool.cc:224] Handling call
0504 20:01:27.195592 (+2183682us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b5400208fd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:27.195706 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1467590400}
I20260504 20:01:27.196820 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.014230 15188 raft_consensus.cc:3055] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:27.079442 15140 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:27.079300 14979 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:27.130396 15136 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.158804 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.019340 15158 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:26.997781 15167 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:26.921082 15048 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:27.083417 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.083602 14994 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.019675 15163 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.013119 15186 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:27.140967 15092 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:27.141088 15045 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.141062 14887 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.017230 15180 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:27.081261 15152 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:27.080832 14807 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:27.152381 15178 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.004s sys 0.014s
I20260504 20:01:27.044018 15157 consensus_queue.cc:1059] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.129s
I20260504 20:01:27.234983 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:27.188063 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 193564us
I20260504 20:01:27.189091 13497 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:27.154826 15151 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:27.154922 15138 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:27.190644 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 742848us
I20260504 20:01:27.169997 14966 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.191998 15032 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.169551 14899 raft_consensus.cc:697] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:27.016153 15189 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.195806 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.198220 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 362) took 1562 ms. Trace:
I20260504 20:01:27.214002 15106 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:27.208091 15007 raft_consensus.cc:697] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:27.208194 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.216079 15163 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.216430 15186 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:27.226316 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:27.230672 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:27.190825 15041 consensus_queue.cc:260] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:27.233180 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 810569us
W20260504 20:01:27.275722 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:27.190246 15108 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:27.236894 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 47948.8us
I20260504 20:01:27.272346 15007 consensus_queue.cc:237] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.206123 15188 raft_consensus.cc:740] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:27.210333 13529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 363) took 1184 ms. Trace:
I20260504 20:01:27.213625 15172 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
W20260504 20:01:27.245106 12716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.003s sys 0.012s
W20260504 20:01:27.283607 12716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.952s user 0.005s sys 0.013s
I20260504 20:01:27.239279 15151 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:27.217864 14891 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:27.231484 15178 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s user 0.005s sys 0.014s
I20260504 20:01:27.264326 12745 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:27.248579 12892 rpcz_store.cc:276] 0504 20:01:25.635832 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.644016 (+ 8184us) service_pool.cc:224] Handling call
0504 20:01:27.197096 (+1553080us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:27.197320 (+ 224us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:27.197342 (+ 22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:27.197385 (+ 43us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:27.197418 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:27.197473 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:27.198031 (+ 558us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54003b9f50. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:27.198152 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3470766976}
W20260504 20:01:27.231155 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 57914.8us
I20260504 20:01:27.267557 15186 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:27.267906 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.268668 15195 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:27.270174 14899 consensus_queue.cc:237] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.230161 14979 raft_consensus.cc:697] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:27.229820 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.273764 12895 raft_consensus.cc:1240] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:27.205158 15140 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:27.209787 15048 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.237918 14807 raft_consensus.cc:697] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:27.284005 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 953393us
W20260504 20:01:27.208854 15009 consensus_meta.cc:287] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.179s user 0.012s sys 0.001s
I20260504 20:01:27.296649 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.250430 15136 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:27.278273 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:27.285362 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 84010.4us
I20260504 20:01:27.284065 15108 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:27.374331 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 108010us
W20260504 20:01:27.320037 12863 connection.cc:254] RPC call timeout handler was delayed by -1.660418408000055s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20779999999999998s for a response
I20260504 20:01:27.377682 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.320762 15140 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:27.381583 15003 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:27.375145 12791 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:27.384196 15003 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.379258 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.386494 15163 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:27.386701 13533 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:27.387315 15163 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:27.280920 13529 rpcz_store.cc:276] 0504 20:01:26.026266 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:26.410281 (+384015us) service_pool.cc:224] Handling call
0504 20:01:27.210260 (+799979us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1920}
W20260504 20:01:27.385435 12716 connection.cc:254] RPC call timeout handler was delayed by -1.344292800000062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20260504 20:01:27.389111 12791 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:27.375773 15136 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:27.387955 15163 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:27.352829 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.390784 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:27.391839 12791 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:27.394021 12791 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:27.442150 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 65996.8us
W20260504 20:01:27.438728 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:27.444137 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 50152.8us
W20260504 20:01:27.450817 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 59910.9us
I20260504 20:01:27.452874 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.455314 12865 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:27.432586 15009 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.319788 15198 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:27.460269 12893 raft_consensus.cc:1240] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:27.441490 15158 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:27.474774 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.318984 15180 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:27.434267 15178 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:27.393427 14887 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:27.432061 15136 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:27.432796 15138 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:27.448675 15045 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:27.485076 12745 raft_consensus.cc:2393] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:27.441123 15196 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:27.453698 15167 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:27.525305 14807 consensus_queue.cc:237] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.533676 15167 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:27.534399 15167 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:27.495146 12716 leader_election.cc:336] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.616s (SENT)
I20260504 20:01:27.538631 15003 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:27.510365 14887 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.440786 15188 consensus_queue.cc:260] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.539808 15032 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:27.541186 15201 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:27.463335 15194 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
W20260504 20:01:27.545500 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:27.476974 15158 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:27.439908 14979 consensus_queue.cc:237] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.554688 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.528832 15138 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:27.579149 15201 raft_consensus.cc:697] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:27.530297 15196 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:27.543345 15045 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.513368 15198 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.580261 15138 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:27.589497 14887 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:27.590550 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:27.594681 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.615058 12745 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:01:27.616166 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.572360 15158 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:27.586529 15090 consensus_meta.cc:287] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.920s user 0.019s sys 0.001s
I20260504 20:01:27.570091 14966 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:27.583320 15108 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:27.596618 15196 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:27.632215 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:27.629673 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 163175us
I20260504 20:01:27.637630 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 304) took 2620 ms. Trace:
I20260504 20:01:27.638252 12749 rpcz_store.cc:276] 0504 20:01:25.017109 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.017670 (+ 561us) service_pool.cc:224] Handling call
0504 20:01:25.018026 (+ 356us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:25.639627 (+621601us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:25.639661 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:25.645189 (+ 5528us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:25.645238 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:27.637314 (+1992076us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b54000dbbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:27.637365 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:27.637536 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1429022592}
I20260504 20:01:27.511797 15178 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.655869 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:27.656806 12749 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:27.661401 14229 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:27.662045 14229 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:27.582772 15201 consensus_queue.cc:237] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.630388 15090 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:27.758589 12890 raft_consensus.cc:1240] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:27.762440 15206 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20260504 20:01:27.762805 15041 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:27.824326 15077 raft_consensus.cc:3055] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:27.825014 15077 raft_consensus.cc:740] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:27.838837 14952 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:27.857556 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:27.858621 13531 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
W20260504 20:01:27.895810 13528 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40924 (request call id 361) took 1870 ms (client timeout 1732 ms). Trace:
W20260504 20:01:27.898814 13528 rpcz_store.cc:269] 0504 20:01:26.025630 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:26.026584 (+ 954us) service_pool.cc:224] Handling call
0504 20:01:27.895455 (+1868871us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540058c354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:27.895677 (+ 222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476520704}
W20260504 20:01:27.909116 15140 consensus_meta.cc:287] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.525s user 0.010s sys 0.019s
W20260504 20:01:27.996366 15186 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.679s user 0.023s sys 0.007s
W20260504 20:01:27.996898 15183 log.cc:927] Time spent T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 1.967s user 0.013s sys 0.012s
W20260504 20:01:27.999099 15150 consensus_meta.cc:287] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.707s user 0.021s sys 0.013s
I20260504 20:01:27.999766 15150 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:27.909706 15140 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:28.019317 15151 consensus_meta.cc:287] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.705s user 0.014s sys 0.001s
W20260504 20:01:28.060102 15142 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 4.493s user 0.029s sys 0.044s
I20260504 20:01:28.309546 12742 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
W20260504 20:01:28.312376 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:28.318653 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:28.320405 12821 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:28.322685 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:28.323891 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:28.163542 15163 consensus_meta.cc:287] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.773s user 0.013s sys 0.003s
I20260504 20:01:27.996973 15186 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:28.203400 15196 consensus_meta.cc:287] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.553s user 0.020s sys 0.008s
I20260504 20:01:28.338907 15073 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:28.342135 14899 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.009s sys 0.024s
I20260504 20:01:28.344136 15189 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:27.964448 14726 consensus_meta.cc:287] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.782s user 0.003s sys 0.026s
W20260504 20:01:28.352705 14807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.010s sys 0.013s
W20260504 20:01:28.470976 14807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s user 0.011s sys 0.014s
I20260504 20:01:28.470597 14726 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:28.472214 12743 raft_consensus.cc:2393] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:28.353184 13497 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.863s user 0.005s sys 0.025s
I20260504 20:01:28.472119 15073 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:28.356702 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 896333us
I20260504 20:01:28.353417 15188 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:28.344314 14891 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:28.374591 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1775) took 3345 ms. Trace:
I20260504 20:01:28.375046 15116 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.058276 15151 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:28.422521 15163 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.422546 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1740) took 1193 ms. Trace:
I20260504 20:01:28.478199 12745 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:28.417224 15142 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 4.950s user 0.032s sys 0.044s
W20260504 20:01:28.418279 15141 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.000s sys 0.010s
W20260504 20:01:28.441315 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:28.453769 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 78120.6us
I20260504 20:01:28.426234 15045 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.354465 15178 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:28.473434 13497 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.993s user 0.006s sys 0.025s
I20260504 20:01:28.486606 12749 rpcz_store.cc:276] 0504 20:01:27.229302 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.321542 (+1092240us) service_pool.cc:224] Handling call
0504 20:01:28.422394 (+100852us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":138368}
I20260504 20:01:28.482834 12865 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:28.475457 14899 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.866s user 0.009s sys 0.024s
I20260504 20:01:28.485307 12748 raft_consensus.cc:2393] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:28.477092 12893 rpcz_store.cc:276] 0504 20:01:25.028258 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.585361 (+557103us) service_pool.cc:224] Handling call
0504 20:01:25.585675 (+ 314us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:27.467801 (+1882126us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:27.829190 (+361389us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54001b8d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:28.374048 (+544858us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4310660992}
I20260504 20:01:28.483582 12819 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:28.502355 14887 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:28.508466 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:28.498685 15141 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.023s user 0.000s sys 0.012s
I20260504 20:01:28.489976 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1733) took 1341 ms. Trace:
I20260504 20:01:28.521852 12745 rpcz_store.cc:276] 0504 20:01:27.145063 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.321545 (+1176482us) service_pool.cc:224] Handling call
0504 20:01:28.486908 (+165363us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17792}
W20260504 20:01:28.522956 12747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57934 (request call id 310) took 2725 ms (client timeout 1577 ms). Trace:
I20260504 20:01:28.523347 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:28.523661 12747 rpcz_store.cc:269] 0504 20:01:25.797693 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.800588 (+ 2895us) service_pool.cc:224] Handling call
0504 20:01:28.522365 (+2721777us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b54002e54d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.522651 (+ 286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2285847936}
I20260504 20:01:28.525867 15163 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.526754 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:28.525501 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:28.485700 15151 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:28.504614 14887 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.554028 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1746) took 1173 ms. Trace:
I20260504 20:01:28.554811 12745 rpcz_store.cc:276] 0504 20:01:27.377071 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.526018 (+1148947us) service_pool.cc:224] Handling call
0504 20:01:28.550456 (+ 24438us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.555496 14899 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election won for term 2
W20260504 20:01:28.564561 12898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1758) took 5409 ms (client timeout 1791 ms). Trace:
I20260504 20:01:28.565169 12747 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 1.
I20260504 20:01:28.550688 15141 raft_consensus.cc:2804] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:28.566318 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1745) took 1264 ms. Trace:
I20260504 20:01:28.575842 12749 raft_consensus.cc:2393] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:28.575313 12747 rpcz_store.cc:276] 0504 20:01:27.301864 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.524903 (+1223039us) service_pool.cc:224] Handling call
0504 20:01:28.566206 (+ 41303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.576521 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1744) took 1279 ms. Trace:
I20260504 20:01:28.566954 12899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1756) took 5423 ms. Trace:
I20260504 20:01:28.587960 12749 rpcz_store.cc:276] 0504 20:01:27.297411 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.507642 (+1210231us) service_pool.cc:224] Handling call
0504 20:01:28.576421 (+ 68779us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.564653 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1777) took 2994 ms. Trace:
W20260504 20:01:28.565502 12898 rpcz_store.cc:269] 0504 20:01:23.154643 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.154819 (+ 176us) service_pool.cc:224] Handling call
0504 20:01:28.563599 (+5408780us) spinlock_profiling.cc:243] Waited 2.82 s on lock 0x7b54003ba954. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.564060 (+ 461us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6489385216}
I20260504 20:01:28.592417 12865 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:28.593878 14891 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:28.595058 15073 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.566697 14899 raft_consensus.cc:697] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:28.600523 12864 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.138s user 0.000s sys 0.013s
W20260504 20:01:28.603189 12864 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.142s user 0.000s sys 0.015s
W20260504 20:01:28.604885 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 1.1527e+06us
I20260504 20:01:28.606729 14726 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:28.607735 15186 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:28.566954 15152 consensus_meta.cc:287] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.345s user 0.027s sys 0.005s
I20260504 20:01:28.587577 12899 rpcz_store.cc:276] 0504 20:01:23.143890 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.146995 (+ 3105us) service_pool.cc:224] Handling call
0504 20:01:24.579863 (+1432868us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:24.618678 (+ 38815us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.566319 (+3947641us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54002085d0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.566679 (+ 360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3222956928}
I20260504 20:01:28.568468 15141 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:28.626051 14887 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.589027 12817 rpcz_store.cc:276] 0504 20:01:25.570140 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.583814 (+ 13674us) service_pool.cc:224] Handling call
0504 20:01:25.584250 (+ 436us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:25.665487 (+ 81237us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:25.665519 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:25.675787 (+ 10268us) log.cc:844] Serialized 26 byte log entry
0504 20:01:25.815177 (+139390us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:25.815219 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:27.869526 (+2054307us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:28.563386 (+693860us) spinlock_profiling.cc:243] Waited 81 ms on lock 0x7b5400375854. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:28.563519 (+ 133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:28.563657 (+ 138us) raft_consensus.cc:1685] finished
0504 20:01:28.563690 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:28.564205 (+ 515us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195312768,"thread_start_us":138968,"threads_started":1,"wal-append.queue_time_us":156634}
I20260504 20:01:28.628237 15141 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:28.650403 15214 log.cc:927] Time spent T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.138s user 0.001s sys 0.003s
I20260504 20:01:28.653901 15196 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.654567 12790 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:28.659531 12817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 239) took 2629 ms (client timeout 2119 ms). Trace:
W20260504 20:01:28.666544 12817 rpcz_store.cc:269] 0504 20:01:26.029506 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.653755 (+2624249us) service_pool.cc:211] Skipping call since client already timed out
0504 20:01:28.658782 (+ 5027us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 20:01:28.671731 15196 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.638415 15140 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:28.678062 15009 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:28.629310 15152 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:28.671837 12850 heartbeater.cc:646] Failed to heartbeat to 127.11.68.62:41865 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.68.62:41865 timed out after 15.000s (SENT)
I20260504 20:01:28.686820 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:28.687891 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:28.682099 15196 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:28.689980 12747 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:28.690891 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1732) took 1546 ms. Trace:
I20260504 20:01:28.691111 12747 rpcz_store.cc:276] 0504 20:01:27.144795 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.585270 (+1440475us) service_pool.cc:224] Handling call
0504 20:01:28.690815 (+105545us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.689980 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:28.693049 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 82465us
W20260504 20:01:28.699221 12817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 236) took 2670 ms (client timeout 2169 ms). Trace:
I20260504 20:01:28.692881 12749 raft_consensus.cc:2410] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:28.637245 14899 consensus_queue.cc:237] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:28.700086 12817 rpcz_store.cc:269] 0504 20:01:26.029006 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.695102 (+2666096us) service_pool.cc:211] Skipping call since client already timed out
0504 20:01:28.699105 (+ 4003us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 20:01:28.700928 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:28.703202 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:28.704479 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1790) took 3110 ms. Trace:
I20260504 20:01:28.706207 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1725) took 1915 ms. Trace:
I20260504 20:01:28.705040 12825 rpcz_store.cc:276] 0504 20:01:25.593416 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:26.019557 (+426141us) service_pool.cc:224] Handling call
0504 20:01:26.019947 (+ 390us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:26.364703 (+344756us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:26.364734 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:26.365213 (+ 479us) log.cc:844] Serialized 26 byte log entry
0504 20:01:26.729632 (+364419us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:26.729713 (+ 81us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.703595 (+1973882us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b540014dcd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:28.703689 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:28.703796 (+ 107us) raft_consensus.cc:1685] finished
0504 20:01:28.703808 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:28.704224 (+ 416us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2697,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":12174,"spinlock_wait_cycles":791894912,"thread_start_us":364009,"threads_started":1,"wal-append.queue_time_us":777526,"wal-append.run_cpu_time_us":2227,"wal-append.run_wall_time_us":1061915}
W20260504 20:01:28.706696 12822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1740) took 5163 ms (client timeout 2078 ms). Trace:
W20260504 20:01:28.703329 13499 connection.cc:254] RPC call timeout handler was delayed by -1.0206392910000659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2169s for a response
I20260504 20:01:28.707372 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1743) took 1432 ms. Trace:
I20260504 20:01:28.706786 12747 rpcz_store.cc:276] 0504 20:01:26.790303 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.704600 (+1914297us) service_pool.cc:224] Handling call
0504 20:01:28.705217 (+ 617us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:28.705710 (+ 493us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:28.705749 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:28.705859 (+ 110us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:28.705911 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.705972 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:28.706108 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.707860 12749 rpcz_store.cc:276] 0504 20:01:27.274939 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.612771 (+1337832us) service_pool.cc:224] Handling call
0504 20:01:28.707304 (+ 94533us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1921536}
I20260504 20:01:28.711374 15152 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.711678 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1725) took 6047 ms. Trace:
I20260504 20:01:28.712327 12824 rpcz_store.cc:276] 0504 20:01:22.664344 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:22.690196 (+ 25852us) service_pool.cc:224] Handling call
0504 20:01:22.743146 (+ 52950us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:25.637464 (+2894318us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.710930 (+3073466us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b54000e38d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:28.711393 (+ 463us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5615250176}
I20260504 20:01:28.699539 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1735) took 1541 ms. Trace:
I20260504 20:01:28.714902 12745 rpcz_store.cc:276] 0504 20:01:27.157971 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.567745 (+1409774us) service_pool.cc:224] Handling call
0504 20:01:28.699444 (+131699us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83072}
W20260504 20:01:28.707623 12822 rpcz_store.cc:269] 0504 20:01:23.543064 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.551747 (+ 8683us) service_pool.cc:224] Handling call
0504 20:01:26.402270 (+2850523us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:28.706261 (+2303991us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b5400398d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.706456 (+ 195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1372856448}
I20260504 20:01:28.719305 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.720562 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 241) took 1267 ms. Trace:
I20260504 20:01:28.720990 12822 rpcz_store.cc:276] 0504 20:01:27.453132 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.718174 (+1265042us) service_pool.cc:224] Handling call
0504 20:01:28.720470 (+ 2296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.726199 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:28.708778 12817 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 237) took 2679 ms (client timeout 2230 ms). Trace:
W20260504 20:01:28.728111 12817 rpcz_store.cc:269] 0504 20:01:26.029181 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.701152 (+2671971us) service_pool.cc:211] Skipping call since client already timed out
0504 20:01:28.708677 (+ 7525us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 20:01:28.730283 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 77630.6us
I20260504 20:01:28.691870 15196 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:28.738149 15196 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:28.749965 12825 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:28.750511 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:28.751108 12825 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 244) took 1256 ms (client timeout 1501 ms). Trace:
I20260504 20:01:28.751581 12824 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
W20260504 20:01:28.751652 12825 rpcz_store.cc:269] 0504 20:01:27.494980 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.719862 (+1224882us) service_pool.cc:224] Handling call
0504 20:01:28.750984 (+ 31122us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:28.752389 12824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 245) took 1256 ms (client timeout 1633 ms). Trace:
I20260504 20:01:28.754170 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.755493 12790 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:28.753110 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.760185 14872 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:28.763206 15041 consensus_meta.cc:287] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.999s user 0.017s sys 0.013s
I20260504 20:01:28.772869 15143 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.773406 15116 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:28.771474 12742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1722) took 2068 ms (client timeout 1951 ms). Trace:
I20260504 20:01:28.772636 15041 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260504 20:01:28.752851 12824 rpcz_store.cc:269] 0504 20:01:27.496947 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.726829 (+1229882us) service_pool.cc:224] Handling call
0504 20:01:28.752303 (+ 25474us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:28.747051 13497 leader_election.cc:336] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.784s (SENT)
I20260504 20:01:28.778312 15116 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.780620 14891 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:28.780829 12748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1728) took 1989 ms (client timeout 1686 ms). Trace:
I20260504 20:01:28.781279 12749 raft_consensus.cc:1240] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:28.782146 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1726) took 1991 ms. Trace:
I20260504 20:01:28.768254 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:28.782822 12749 rpcz_store.cc:276] 0504 20:01:26.790636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.727524 (+1936888us) service_pool.cc:224] Handling call
0504 20:01:28.749410 (+ 21886us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:28.781838 (+ 32428us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.782043 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:28.781416 12748 rpcz_store.cc:269] 0504 20:01:26.791221 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:27.012759 (+221538us) service_pool.cc:224] Handling call
0504 20:01:28.780538 (+1767779us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54002bd9d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.780738 (+ 200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3267175680}
I20260504 20:01:28.784866 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 313) took 1520 ms. Trace:
I20260504 20:01:28.786029 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 314) took 1517 ms. Trace:
I20260504 20:01:28.786527 12748 rpcz_store.cc:276] 0504 20:01:27.268252 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.784729 (+1516477us) service_pool.cc:224] Handling call
0504 20:01:28.785157 (+ 428us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:28.785534 (+ 377us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:28.785572 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:28.785650 (+ 78us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:28.785725 (+ 75us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.785820 (+ 95us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:28.785952 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:28.787662 12750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1715) took 7031 ms (client timeout 1933 ms). Trace:
W20260504 20:01:28.788188 12750 rpcz_store.cc:269] 0504 20:01:21.756217 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:21.756367 (+ 150us) service_pool.cc:224] Handling call
0504 20:01:28.787165 (+7030798us) spinlock_profiling.cc:243] Waited 3.76 s on lock 0x7b54003d9d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.787427 (+ 262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8658088832}
W20260504 20:01:28.790477 12746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1704) took 8306 ms (client timeout 1720 ms). Trace:
W20260504 20:01:28.790983 12746 rpcz_store.cc:269] 0504 20:01:20.484014 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:20.486008 (+ 1994us) service_pool.cc:224] Handling call
0504 20:01:28.790089 (+8304081us) spinlock_profiling.cc:243] Waited 4.47 s on lock 0x7b5400243454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.790268 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10285841664}
I20260504 20:01:28.786875 12749 rpcz_store.cc:276] 0504 20:01:27.264997 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.783731 (+1518734us) service_pool.cc:224] Handling call
0504 20:01:28.784139 (+ 408us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:28.784506 (+ 367us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:28.784530 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:28.784586 (+ 56us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:28.784630 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.784693 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:28.784812 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:28.791906 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 50580.6us
I20260504 20:01:28.793197 12825 raft_consensus.cc:2410] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260504 20:01:28.797376 12750 raft_consensus.cc:1240] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:28.798889 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1743) took 1429 ms. Trace:
I20260504 20:01:28.800100 12822 rpcz_store.cc:276] 0504 20:01:27.369253 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.721789 (+1352536us) service_pool.cc:224] Handling call
0504 20:01:28.798801 (+ 77012us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.781853 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:28.801998 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 54103.5us
I20260504 20:01:28.802441 12824 raft_consensus.cc:2393] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:28.803012 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1750) took 1312 ms. Trace:
I20260504 20:01:28.803391 12824 rpcz_store.cc:276] 0504 20:01:27.490939 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.781374 (+1290435us) service_pool.cc:224] Handling call
0504 20:01:28.802921 (+ 21547us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.804540 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:28.805657 12824 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:28.811017 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 55660.2us
I20260504 20:01:28.797454 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 243) took 1312 ms. Trace:
I20260504 20:01:28.811856 12825 rpcz_store.cc:276] 0504 20:01:27.484925 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.752687 (+1267762us) service_pool.cc:224] Handling call
0504 20:01:28.797372 (+ 44685us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.806704 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 311) took 1611 ms. Trace:
I20260504 20:01:28.813982 12747 rpcz_store.cc:276] 0504 20:01:27.195374 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.741474 (+1546100us) service_pool.cc:224] Handling call
0504 20:01:28.779212 (+ 37738us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:28.806213 (+ 27001us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:28.806257 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:28.806347 (+ 90us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:28.806406 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.806482 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:28.806640 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:28.815035 12818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1788) took 3217 ms (client timeout 2096 ms). Trace:
I20260504 20:01:28.811679 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:28.816022 15136 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.271s user 0.019s sys 0.013s
I20260504 20:01:28.816133 12824 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:28.816641 15136 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:28.817248 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.818120 12824 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:28.818455 12890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1774) took 3790 ms. Trace:
I20260504 20:01:28.819094 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.819011 12890 rpcz_store.cc:276] 0504 20:01:25.027911 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.100195 (+ 72284us) service_pool.cc:224] Handling call
0504 20:01:25.554105 (+453910us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:27.759103 (+2204998us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.817828 (+1058725us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54001faed4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:28.818241 (+ 413us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5069298816}
I20260504 20:01:28.819907 12824 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
W20260504 20:01:28.807585 12820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1786) took 3214 ms (client timeout 2372 ms). Trace:
I20260504 20:01:28.808297 15140 consensus_queue.cc:1059] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:28.801453 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:28.793040 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1789) took 3199 ms. Trace:
I20260504 20:01:28.813114 12790 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:28.813153 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:28.774260 12742 rpcz_store.cc:269] 0504 20:01:26.703030 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:27.063872 (+360842us) service_pool.cc:224] Handling call
0504 20:01:28.771147 (+1707275us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400242a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.771358 (+ 211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2860149632}
I20260504 20:01:28.821189 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:28.815636 12818 rpcz_store.cc:269] 0504 20:01:25.592840 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.738072 (+145232us) service_pool.cc:224] Handling call
0504 20:01:28.809558 (+3071486us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b540015dbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.809776 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":968871296}
I20260504 20:01:28.827546 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.835233 12822 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:28.836146 15138 consensus_meta.cc:287] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.217s user 0.024s sys 0.003s
I20260504 20:01:28.836786 15138 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:28.822404 12820 rpcz_store.cc:269] 0504 20:01:25.592261 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.845606 (+253345us) service_pool.cc:224] Handling call
0504 20:01:28.806867 (+2961261us) spinlock_profiling.cc:243] Waited 71.3 ms on lock 0x7b54000e47d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.807081 (+ 214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":163907840}
I20260504 20:01:28.824986 12816 rpcz_store.cc:276] 0504 20:01:25.593114 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:26.019484 (+426370us) service_pool.cc:224] Handling call
0504 20:01:26.049870 (+ 30386us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:26.232825 (+182955us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:26.232863 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:26.232908 (+ 45us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:26.232938 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.792499 (+2559561us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54000e4cd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:28.792573 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:28.792813 (+ 240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":420203776}
I20260504 20:01:28.847096 14872 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:01:28.850435 14229 consensus_meta.cc:287] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.061s user 0.038s sys 0.020s
I20260504 20:01:28.856860 14229 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.863840 12824 raft_consensus.cc:2393] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:28.853238 12825 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
W20260504 20:01:28.870038 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 57083.4us
I20260504 20:01:28.874055 15152 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:28.874615 15152 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:28.856043 13497 leader_election.cc:336] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.169s (SENT)
I20260504 20:01:28.875047 15152 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:28.876304 13497 leader_election.cc:336] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.230s (SENT)
I20260504 20:01:28.878043 15119 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:28.880963 15179 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:28.881405 14229 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.886158 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.887480 12818 raft_consensus.cc:2393] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:28.888540 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:28.888979 14952 consensus_meta.cc:287] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.049s user 0.003s sys 0.022s
I20260504 20:01:28.889248 12818 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:28.889829 15161 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:28.890290 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.890473 15161 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.891191 12820 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:28.891964 12741 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:28.892300 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.892709 15090 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:28.893018 12820 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
W20260504 20:01:28.893388 12741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1579) took 5669 ms (client timeout 1933 ms). Trace:
W20260504 20:01:28.893877 12741 rpcz_store.cc:269] 0504 20:01:23.223761 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.224634 (+ 873us) service_pool.cc:224] Handling call
0504 20:01:28.893134 (+5668500us) spinlock_profiling.cc:243] Waited 5.66 s on lock 0x7b540007bcd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.893304 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13026729856}
I20260504 20:01:28.894526 15048 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:28.853049 15158 consensus_meta.cc:287] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.222s user 0.007s sys 0.019s
I20260504 20:01:28.895534 15158 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.896862 12715 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:28.898141 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:28.898509 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.898921 12816 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:28.900272 12825 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:28.901257 12825 raft_consensus.cc:606] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:28.881748 15179 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.906908 15150 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:28.903697 15152 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:28.910152 15152 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:28.920282 15094 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:28.920979 15094 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:28.921082 15138 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:28.921424 15094 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:28.922289 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 116599us
I20260504 20:01:28.923638 12825 raft_consensus.cc:614] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:28.928938 12817 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:28.907668 15179 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:28.930438 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1753) took 1374 ms. Trace:
W20260504 20:01:28.855921 15167 consensus_meta.cc:287] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.321s user 0.019s sys 0.015s
I20260504 20:01:28.934141 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49576 (request call id 32) took 2582 ms. Trace:
I20260504 20:01:28.934861 12825 rpcz_store.cc:276] 0504 20:01:26.342644 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.899944 (+2557300us) service_pool.cc:224] Handling call
0504 20:01:28.924512 (+ 24568us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.936582 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:28.943091 12820 raft_consensus.cc:2393] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:28.933529 12824 rpcz_store.cc:276] 0504 20:01:27.556409 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.820862 (+1264453us) service_pool.cc:224] Handling call
0504 20:01:28.930339 (+109477us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.930107 14872 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:28.952508 14872 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:28.922540 15077 consensus_queue.cc:260] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:28.954173 15077 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:28.954394 12865 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:28.881122 15119 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:28.975517 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:28.962620 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:28.933434 15158 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:29.018646 12823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1779) took 3401 ms (client timeout 1628 ms). Trace:
I20260504 20:01:29.007047 15082 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:29.020412 15082 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.014438 15138 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:29.022627 15138 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.024269 12791 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:29.024747 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1770) took 4012 ms. Trace:
I20260504 20:01:29.014622 15152 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:29.019207 12823 rpcz_store.cc:269] 0504 20:01:25.588684 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.589201 (+ 517us) service_pool.cc:224] Handling call
0504 20:01:28.975388 (+3386187us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54003002d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:28.989929 (+ 14541us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":371276544}
I20260504 20:01:29.026472 15152 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.027437 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.028191 12823 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:29.028573 15152 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.029157 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.029722 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 234) took 3003 ms. Trace:
W20260504 20:01:29.030010 15228 log.cc:927] Time spent T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.066s user 0.004s sys 0.000s
I20260504 20:01:29.030201 12820 rpcz_store.cc:276] 0504 20:01:26.028460 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.983849 (+2955389us) service_pool.cc:224] Handling call
0504 20:01:29.028997 (+ 45148us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.029300 (+ 303us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:29.029329 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:29.029399 (+ 70us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:29.029450 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.029512 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.029644 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.033270 15138 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:29.034574 15175 log.cc:927] Time spent T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 3.005s user 0.017s sys 0.011s
I20260504 20:01:29.036068 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:29.031821 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.038635 15150 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:29.039249 15150 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:29.025218 12895 rpcz_store.cc:276] 0504 20:01:25.012683 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.021748 (+ 9065us) service_pool.cc:224] Handling call
0504 20:01:26.408943 (+1387195us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3a3e 000000000035a3c9 00007f016190b8e0 00007f01619d4784 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:26.409111 (+ 168us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:27.318370 (+909259us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.024253 (+1705883us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54002f67d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:29.024543 (+ 290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3011298944}
I20260504 20:01:29.001593 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 232) took 2975 ms. Trace:
I20260504 20:01:29.041693 12818 rpcz_store.cc:276] 0504 20:01:26.025907 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.934061 (+2908154us) service_pool.cc:224] Handling call
0504 20:01:28.999048 (+ 64987us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.000211 (+ 1163us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:29.000284 (+ 73us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:29.000485 (+ 201us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:29.000640 (+ 155us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.000883 (+ 243us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.001322 (+ 439us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.041742 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 238) took 2989 ms. Trace:
I20260504 20:01:29.039754 15150 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:29.044281 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.045087 15119 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.045367 12741 raft_consensus.cc:2410] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:29.046074 13531 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:29.046447 15140 raft_consensus.cc:2804] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:29.046892 15140 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:29.047312 15140 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:29.048364 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 233) took 3022 ms. Trace:
I20260504 20:01:29.048880 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:29.050487 12821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 231) took 3024 ms (client timeout 1542 ms). Trace:
W20260504 20:01:29.051028 12821 rpcz_store.cc:269] 0504 20:01:26.025618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:27.001826 (+976208us) service_pool.cc:224] Handling call
0504 20:01:29.040267 (+2038441us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540039a154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:29.050338 (+ 10071us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3027230080}
I20260504 20:01:29.052335 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.053128 12821 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:29.052345 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:29.058056 12819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 230) took 3032 ms (client timeout 1784 ms). Trace:
I20260504 20:01:29.058658 15082 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:29.058558 12819 rpcz_store.cc:269] 0504 20:01:26.025290 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:27.118903 (+1093613us) service_pool.cc:224] Handling call
0504 20:01:29.057730 (+1938827us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54000e33d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:29.057955 (+ 225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3024903936}
I20260504 20:01:29.059209 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 242) took 1589 ms. Trace:
I20260504 20:01:29.059623 12817 rpcz_store.cc:276] 0504 20:01:27.470452 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.730041 (+1259589us) service_pool.cc:224] Handling call
0504 20:01:29.058991 (+328950us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b54000e4cd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:29.059107 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":69930240}
I20260504 20:01:29.060786 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.060910 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.061717 12817 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:29.048774 12824 rpcz_store.cc:276] 0504 20:01:26.026218 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.985120 (+2958902us) service_pool.cc:224] Handling call
0504 20:01:29.047686 (+ 62566us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.047989 (+ 303us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:29.048016 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:29.048081 (+ 65us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:29.048137 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.048193 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.048299 (+ 106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.961364 15136 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.090265 15116 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.091881 12894 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1776) took 3984 ms. Trace:
I20260504 20:01:29.092442 12894 rpcz_store.cc:276] 0504 20:01:25.108467 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:25.584508 (+476041us) service_pool.cc:224] Handling call
0504 20:01:25.645360 (+ 60852us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:27.033199 (+1387839us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.073742 (+2040543us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b54001fb3d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:29.091625 (+ 17883us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1761109248}
I20260504 20:01:29.095969 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.098834 12823 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:29.099512 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:29.043011 12825 rpcz_store.cc:276] 0504 20:01:26.029336 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.002696 (+2973360us) service_pool.cc:224] Handling call
0504 20:01:29.003980 (+ 1284us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.004843 (+ 863us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:29.004940 (+ 97us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:29.005150 (+ 210us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:29.005334 (+ 184us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.005554 (+ 220us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.018274 (+ 12720us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:28.951246 15167 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.103001 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1734) took 1933 ms. Trace:
W20260504 20:01:29.072468 15224 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.218s user 0.001s sys 0.000s
I20260504 20:01:29.106633 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.061623 13525 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.101673 12791 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.065382 12863 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.116302 12742 raft_consensus.cc:1275] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:29.116480 13526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 544) took 5988 ms. Trace:
I20260504 20:01:29.116977 13526 rpcz_store.cc:276] 0504 20:01:23.127918 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:23.135424 (+ 7506us) service_pool.cc:224] Handling call
0504 20:01:23.162994 (+ 27570us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:25.209381 (+2046387us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:25.231909 (+ 22528us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:25.271040 (+ 39131us) log.cc:844] Serialized 26 byte log entry
0504 20:01:25.739360 (+468320us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:25.745599 (+ 6239us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.033499 (+3287900us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:29.115815 (+ 82316us) spinlock_profiling.cc:243] Waited 71.1 ms on lock 0x7b540057f8d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:29.115890 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:29.115955 (+ 65us) raft_consensus.cc:1685] finished
0504 20:01:29.115971 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.116241 (+ 270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172125824,"thread_start_us":449876,"threads_started":1,"wal-append.queue_time_us":555360}
I20260504 20:01:29.118937 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.119359 12816 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:29.120538 12822 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:29.113322 12820 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:29.063452 14899 raft_consensus.cc:3055] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:29.123502 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.123342 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.126407 12816 raft_consensus.cc:2393] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:29.125392 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.106025 12745 rpcz_store.cc:276] 0504 20:01:27.154557 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:28.716745 (+1562188us) service_pool.cc:224] Handling call
0504 20:01:28.717409 (+ 664us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:28.719491 (+ 2082us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:28.719525 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:28.721157 (+ 1632us) log.cc:844] Serialized 26 byte log entry
0504 20:01:28.938742 (+217585us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:28.939086 (+ 344us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:28.939534 (+ 448us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:29.076149 (+136615us) raft_consensus.cc:1685] finished
0504 20:01:29.076184 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.088079 (+ 11895us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":184,"raft.run_cpu_time_us":739,"raft.run_wall_time_us":738,"thread_start_us":206578,"threads_started":1,"wal-append.queue_time_us":292427}
I20260504 20:01:29.133071 12819 raft_consensus.cc:2393] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:29.136310 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.144124 12825 raft_consensus.cc:2393] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:29.103749 15141 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:29.157229 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 202983us
I20260504 20:01:29.158231 15179 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:29.062626 15090 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:29.167764 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 102655us
I20260504 20:01:29.181432 15092 consensus_queue.cc:1059] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260504 20:01:29.182649 12863 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:29.170182 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 68648.4us
I20260504 20:01:29.173457 15167 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.173581 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.176492 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:29.177311 15161 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.066854 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:29.178354 12743 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1730) took 2144 ms (client timeout 1870 ms). Trace:
I20260504 20:01:29.065716 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.068902 12648 catalog_manager.cc:5671] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:29.095995 15198 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.183610 15092 raft_consensus.cc:3055] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:29.152096 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.181232 15090 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.191651 12899 raft_consensus.cc:1240] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:29.192235 15025 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:01:29.188773 12743 rpcz_store.cc:269] 0504 20:01:27.034236 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:27.144381 (+110145us) service_pool.cc:224] Handling call
0504 20:01:29.112212 (+1967831us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400028d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:29.178241 (+ 66029us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3049648384}
I20260504 20:01:29.195453 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1744) took 1825 ms. Trace:
I20260504 20:01:29.214442 12816 rpcz_store.cc:276] 0504 20:01:27.369556 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.184474 (+1814918us) service_pool.cc:224] Handling call
0504 20:01:29.194405 (+ 9931us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.194921 (+ 516us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:29.194962 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:29.195052 (+ 90us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:29.195116 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.195198 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.195346 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:29.187410 15231 log.cc:927] Time spent T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
W20260504 20:01:29.218024 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 178320us
I20260504 20:01:29.218595 15090 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:29.184005 15179 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:29.213954 15025 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:29.219235 15179 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.219623 15025 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.198355 15141 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:29.200170 12865 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.202775 12715 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.134877 14899 raft_consensus.cc:740] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.208618 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.063052 14823 consensus_queue.cc:1048] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20260504 20:01:29.212441 12817 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:29.215830 13499 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:29.221247 12819 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
W20260504 20:01:29.223007 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:29.229370 12823 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:29.210187 15092 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.236179 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1745) took 1840 ms. Trace:
I20260504 20:01:29.234810 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:29.237706 12819 rpcz_store.cc:276] 0504 20:01:27.395526 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.185979 (+1790453us) service_pool.cc:224] Handling call
0504 20:01:29.220962 (+ 34983us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.235920 (+ 14958us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.236111 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.239144 12863 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:29.240453 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.236696 12825 raft_consensus.cc:1240] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:29.228539 15161 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:29.242201 12865 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:29.241194 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.241187 12823 raft_consensus.cc:2393] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:29.246224 15090 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:29.251435 15090 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:29.251957 15090 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:29.252815 12865 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.249107 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:53094 (request call id 524) took 2113 ms. Trace:
I20260504 20:01:29.244064 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1748) took 1763 ms. Trace:
I20260504 20:01:29.254860 12817 raft_consensus.cc:1240] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:29.254796 12822 rpcz_store.cc:276] 0504 20:01:27.122339 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.151223 (+2028884us) service_pool.cc:224] Handling call
0504 20:01:29.235587 (+ 84364us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.255046 12818 rpcz_store.cc:276] 0504 20:01:27.481261 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.224669 (+1743408us) service_pool.cc:224] Handling call
0504 20:01:29.243425 (+ 18756us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.243711 (+ 286us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:29.243737 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:29.243797 (+ 60us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:29.243837 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.243887 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.243986 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.255831 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1749) took 1767 ms. Trace:
I20260504 20:01:29.256762 15141 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:29.256892 15198 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:29.258001 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1746) took 1793 ms. Trace:
I20260504 20:01:29.257227 14823 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:29.257412 15198 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:29.258813 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.259184 15198 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.259970 12745 raft_consensus.cc:2410] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:29.259892 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.260982 12741 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:29.243721 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.241753 14899 consensus_queue.cc:260] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.263387 14979 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:29.258167 15141 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:29.257061 15092 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.264658 15141 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:29.265542 15092 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.239910 15094 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.262763 12648 catalog_manager.cc:5671] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:29.271173 15094 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:29.275115 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.248634 12824 raft_consensus.cc:2376] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5. Re-sending same reply.
I20260504 20:01:29.256276 12817 rpcz_store.cc:276] 0504 20:01:27.488698 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.233847 (+1745149us) service_pool.cc:224] Handling call
0504 20:01:29.254715 (+ 20868us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.255323 (+ 608us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.255764 (+ 441us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.279676 12648 catalog_manager.cc:5671] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:01:29.274274 12716 leader_election.cc:336] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.628s (SENT)
I20260504 20:01:29.277300 12825 rpcz_store.cc:276] 0504 20:01:27.455611 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.209897 (+1754286us) service_pool.cc:224] Handling call
0504 20:01:29.236499 (+ 26602us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.246854 (+ 10355us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.249009 (+ 2155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.281499 12648 catalog_manager.cc:5671] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: config changed from index 3 to 5, term changed from 3 to 5, VOTER 732c3560a688494196d22a1a48f89968 (127.11.68.3) evicted. New cstate: current_term: 5 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:29.291256 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.292189 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.274325 14899 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:29.294124 12819 raft_consensus.cc:1275] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260504 20:01:29.292073 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.299631 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1751) took 1786 ms. Trace:
I20260504 20:01:29.295245 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.297070 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.300491 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:29.302452 15196 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.508s user 0.020s sys 0.002s
I20260504 20:01:29.303059 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.306066 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.309295 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.300232 12819 rpcz_store.cc:276] 0504 20:01:27.513612 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.247800 (+1734188us) service_pool.cc:224] Handling call
0504 20:01:29.276948 (+ 29148us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.299095 (+ 22147us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.299528 (+ 433us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.321362 12817 raft_consensus.cc:2376] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4. Re-sending same reply.
W20260504 20:01:29.322871 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 83809.6us
I20260504 20:01:29.319109 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.324731 12823 raft_consensus.cc:2368] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
W20260504 20:01:29.330964 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 208458us
I20260504 20:01:29.324631 15196 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.332651 12745 raft_consensus.cc:2393] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:29.311745 12824 raft_consensus.cc:2393] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:29.305073 12791 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:29.337479 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.340744 15094 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:29.341308 15094 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.342837 12816 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:29.351202 12825 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:29.353262 12819 raft_consensus.cc:1240] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:29.354153 15106 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:29.356546 12818 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:29.335179 12648 catalog_manager.cc:5671] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:29.357796 12822 raft_consensus.cc:1275] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:29.359913 12821 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
W20260504 20:01:29.335886 15232 log.cc:927] Time spent T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260504 20:01:29.361590 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1747) took 1887 ms. Trace:
I20260504 20:01:29.362015 12816 rpcz_store.cc:276] 0504 20:01:27.465097 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.217357 (+1752260us) service_pool.cc:224] Handling call
0504 20:01:29.248655 (+ 31298us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:29.248873 (+ 218us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.343222 (+ 94349us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.351810 (+ 8588us) spinlock_profiling.cc:243] Waited 93.8 ms on lock 0x7b54000e42d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:29.351992 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223172608}
I20260504 20:01:29.332158 15208 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:29.363178 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1757) took 1037 ms. Trace:
I20260504 20:01:29.327934 15082 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:29.363941 15216 consensus_queue.cc:1059] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:29.367177 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1752) took 1838 ms. Trace:
W20260504 20:01:29.365571 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 82782.3us
I20260504 20:01:29.366171 12819 rpcz_store.cc:276] 0504 20:01:28.323271 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.328494 (+1005223us) service_pool.cc:224] Handling call
0504 20:01:29.353122 (+ 24628us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.353662 (+ 540us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.360844 (+ 7182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.367810 12825 rpcz_store.cc:276] 0504 20:01:27.528987 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.308833 (+1779846us) service_pool.cc:224] Handling call
0504 20:01:29.350931 (+ 42098us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.363279 (+ 12348us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.367081 (+ 3802us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.371570 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.334311 13525 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:01:29.375097 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 98937.7us
I20260504 20:01:29.379560 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.381628 13532 raft_consensus.cc:2393] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:29.381873 12819 raft_consensus.cc:1240] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:29.386209 13531 raft_consensus.cc:1240] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:29.386284 12716 leader_election.cc:336] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.565s (SENT)
I20260504 20:01:29.322513 14872 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.386971 12716 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.367946 15208 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:29.392175 12790 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:29.395864 15119 raft_consensus.cc:3055] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:29.396855 12816 raft_consensus.cc:1240] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:29.374574 14979 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:29.372339 15196 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:29.402281 12821 raft_consensus.cc:1240] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260504 20:01:29.408952 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 246) took 1862 ms. Trace:
I20260504 20:01:29.402922 15188 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:29.400274 15119 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:29.403259 12791 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:29.373725 15198 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.406376 15077 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:29.409543 12817 rpcz_store.cc:276] 0504 20:01:27.546189 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.322590 (+1776401us) service_pool.cc:224] Handling call
0504 20:01:29.369597 (+ 47007us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.375980 (+ 6383us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:29.376007 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:29.408646 (+ 32639us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:29.408715 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.408774 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.408884 (+ 110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.409715 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.418466 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:29.419025 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.422685 12822 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:29.423708 12816 raft_consensus.cc:1240] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:29.417755 12648 catalog_manager.cc:5671] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:29.401651 15208 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:29.426939 12821 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8. Ops: []
I20260504 20:01:29.429756 12822 raft_consensus.cc:1275] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:29.427791 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.404312 14979 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:29.433856 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.441962 12816 raft_consensus.cc:1240] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:29.441989 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.439656 12863 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:29.442787 12822 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:29.443248 12817 raft_consensus.cc:1240] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:29.450601 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:29.452165 12823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1760) took 1100 ms. Trace:
I20260504 20:01:29.452533 12823 rpcz_store.cc:276] 0504 20:01:28.369767 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.336411 (+966644us) service_pool.cc:224] Handling call
0504 20:01:29.336835 (+ 424us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.354492 (+ 17657us) raft_consensus.cc:1534] Early marking committed up to index 3
0504 20:01:29.354843 (+ 351us) log.cc:844] Serialized 14 byte log entry
0504 20:01:29.451809 (+ 96966us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:29.451852 (+ 43us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:29.451895 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.451976 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:29.452105 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":276,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"spinlock_wait_cycles":13056,"thread_start_us":92190,"threads_started":1}
I20260504 20:01:29.439133 14979 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:29.437006 15106 raft_consensus.cc:2764] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260504 20:01:29.439452 15077 raft_consensus.cc:697] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:29.441020 15094 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.439247 15167 raft_consensus.cc:2804] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260504 20:01:29.457994 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 98330us
I20260504 20:01:29.458189 12821 raft_consensus.cc:1275] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260504 20:01:29.461476 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 70541.6us
I20260504 20:01:29.451171 15188 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:29.437661 15179 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.461337 12863 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:29.468778 15210 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:29.465957 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 168790us
W20260504 20:01:29.463964 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 77736.1us
I20260504 20:01:29.462929 15167 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:29.477314 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:29.477774 15201 consensus_queue.cc:1059] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260504 20:01:29.481657 12791 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.482391 15201 raft_consensus.cc:3055] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:29.478128 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.483403 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.485230 15138 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:29.485684 15138 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:29.486003 15158 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:29.486126 15138 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.482800 15201 raft_consensus.cc:740] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.500777 12899 raft_consensus.cc:1240] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:29.465481 15188 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:29.502503 15106 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:29.465068 15216 consensus_queue.cc:1059] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260504 20:01:29.511713 12822 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:29.510203 12716 leader_election.cc:336] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.609s (SENT)
I20260504 20:01:29.510095 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.518181 12716 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.518842 12821 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.519917 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.476325 15210 raft_consensus.cc:697] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.524767 12714 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:29.488919 15167 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:29.519776 15106 raft_consensus.cc:697] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.532555 15116 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:29.533255 15116 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:29.533600 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.534099 15111 raft_consensus.cc:3055] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:29.534291 12825 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:29.534567 15111 raft_consensus.cc:740] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:29.533633 15116 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.535586 15111 consensus_queue.cc:260] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.536912 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.537633 12745 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:29.528247 15161 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:29.539692 14872 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:29.539963 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.540676 13525 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.507014 15179 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:29.520869 15188 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.541278 12648 catalog_manager.cc:5671] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:29.548341 12817 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:29.541188 15090 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.552500 12863 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:29.513159 15077 consensus_queue.cc:237] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.478674 15119 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.554116 12648 catalog_manager.cc:5671] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 1 to 2, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:01:29.504880 15025 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.562222 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.564013 15179 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:29.564291 15106 consensus_queue.cc:237] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.565122 15210 consensus_queue.cc:237] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.568956 14979 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.506589 15201 consensus_queue.cc:260] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.570674 15201 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.571236 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53094 (request call id 525) took 2448 ms. Trace:
I20260504 20:01:29.571492 15179 raft_consensus.cc:697] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:29.574846 15140 consensus_meta.cc:287] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.527s user 0.013s sys 0.000s
I20260504 20:01:29.573856 15141 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.578305 12745 raft_consensus.cc:2410] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:29.606688 12898 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:29.576020 12819 rpcz_store.cc:276] 0504 20:01:27.122717 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.450675 (+2327958us) service_pool.cc:224] Handling call
0504 20:01:29.571136 (+120461us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:29.547420 15198 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.608296 12863 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:29.615684 12716 leader_election.cc:336] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.757s (SENT)
I20260504 20:01:29.616283 12716 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:29.575479 15179 consensus_queue.cc:237] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:29.619007 12716 leader_election.cc:336] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.780s (SENT)
W20260504 20:01:29.620338 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:29.619715 12716 leader_election.cc:336] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.791s (SENT)
W20260504 20:01:29.622932 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 62567us
I20260504 20:01:29.620196 12714 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:29.625358 14872 consensus_queue.cc:1048] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:29.626163 12824 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:29.630280 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 68470.6us
I20260504 20:01:29.575348 15140 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.633231 14979 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.635149 12821 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
W20260504 20:01:29.635528 15150 consensus_meta.cc:287] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.592s user 0.012s sys 0.000s
I20260504 20:01:29.636059 15150 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.637218 12818 raft_consensus.cc:1240] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:29.639396 12820 raft_consensus.cc:1240] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:29.642096 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 81351us
I20260504 20:01:29.644977 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1763) took 1152 ms. Trace:
I20260504 20:01:29.658814 15161 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:29.659405 15161 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:29.659840 15161 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.646502 15025 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:29.625389 15094 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:29.663868 15090 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:29.665879 15140 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:29.668206 12791 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:29.672127 12863 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:29.627761 15141 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:29.674446 15141 consensus_queue.cc:1059] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:01:29.676064 15141 consensus_queue.cc:1059] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260504 20:01:29.677141 15141 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:01:29.638986 15150 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:29.678412 12716 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.032s (SENT) [suppressed 50 similar messages]
I20260504 20:01:29.679512 14979 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:29.680183 14979 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:29.680689 14979 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:29.619704 15138 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.682209 15106 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:29.682982 15106 raft_consensus.cc:697] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.683621 15138 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:29.684243 15106 consensus_queue.cc:237] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.686028 15009 raft_consensus.cc:3055] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:29.686319 14823 raft_consensus.cc:3055] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:29.686990 14823 raft_consensus.cc:740] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.688855 15111 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:29.688319 14823 consensus_queue.cc:260] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.645485 12818 rpcz_store.cc:276] 0504 20:01:28.489318 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.365297 (+875979us) service_pool.cc:224] Handling call
0504 20:01:29.636985 (+271688us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.637650 (+ 665us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.638864 (+ 1214us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b5400370d50. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:29.638975 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":605939968}
I20260504 20:01:29.686520 15009 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.690981 15119 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:29.694097 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1742) took 2416 ms. Trace:
I20260504 20:01:29.694517 12820 rpcz_store.cc:276] 0504 20:01:27.277841 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.143948 (+1866107us) service_pool.cc:224] Handling call
0504 20:01:29.196458 (+ 52510us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:29.642030 (+445572us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:29.693744 (+ 51714us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b540015dbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:29.693988 (+ 244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1018048256}
I20260504 20:01:29.701938 14823 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:29.703842 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.703750 15009 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.704772 12743 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:29.705271 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.705492 15009 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.706004 12892 raft_consensus.cc:1240] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:29.705875 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:29.709851 12746 raft_consensus.cc:2393] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:29.708678 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:29.714442 12742 raft_consensus.cc:1275] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:29.721565 15090 consensus_queue.cc:1059] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260504 20:01:29.722909 12647 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39332 (request call id 223) took 17026 ms (client timeout 15000 ms). Trace:
I20260504 20:01:29.722990 15090 raft_consensus.cc:3055] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:29.723685 15090 raft_consensus.cc:740] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:29.724994 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:29.724824 15090 consensus_queue.cc:260] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.738076 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:29.738024 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.737614 13525 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:29.771446 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:29.774847 12741 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.776645 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.779502 12790 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:29.780012 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:29.734390 15210 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:29.796700 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.806222 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:29.794291 15141 consensus_queue.cc:1059] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260504 20:01:29.836860 15141 raft_consensus.cc:3055] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:29.837530 15141 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.837759 12741 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:29.838156 15140 consensus_queue.cc:1048] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260504 20:01:29.814477 15198 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:29.841730 15198 raft_consensus.cc:697] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:29.843456 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.843289 15198 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:29.846248 12741 raft_consensus.cc:1240] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:29.846524 12750 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:29.852747 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.853752 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.854131 12820 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:29.857527 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260504 20:01:29.857504 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.859036 12820 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
W20260504 20:01:29.859452 14899 consensus_meta.cc:287] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.534s user 0.015s sys 0.000s
I20260504 20:01:29.860518 13529 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:29.860821 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:01:29.863356 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:29.864956 15090 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.837685 12818 raft_consensus.cc:1275] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:29.867626 15167 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.871011 14979 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:29.723567 12647 rpcz_store.cc:269] 0504 20:01:12.695842 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:12.697440 (+ 1598us) service_pool.cc:224] Handling call
0504 20:01:29.722615 (+17025175us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:21.877469 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:22.035568 (+158099us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:22.093538 (+ 57970us) write_op.cc:435] Acquired schema lock
0504 20:01:22.099528 (+ 5990us) tablet.cc:662] Decoding operations
0504 20:01:22.507754 (+408226us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:22.507873 (+ 119us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:22.507928 (+ 55us) tablet.cc:685] Acquiring locks for 8 operations
0504 20:01:22.509202 (+ 1274us) tablet.cc:701] Row locks acquired
0504 20:01:22.509224 (+ 22us) write_op.cc:260] PREPARE: finished
0504 20:01:22.509514 (+ 290us) write_op.cc:270] Start()
0504 20:01:22.509711 (+ 197us) write_op.cc:276] Timestamp: P: 1777924882509417 usec, L: 0
0504 20:01:22.509737 (+ 26us) op_driver.cc:348] REPLICATION: starting
0504 20:01:22.511006 (+ 1269us) log.cc:844] Serialized 3205 byte log entry
0504 20:01:25.810912 (+3299906us) op_driver.cc:464] REPLICATION: finished
0504 20:01:26.514527 (+703615us) write_op.cc:301] APPLY: starting
0504 20:01:26.614825 (+100298us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:27.063722 (+448897us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:27.082239 (+ 18517us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:27.196176 (+113937us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:27.196404 (+ 228us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0504 20:01:27.196459 (+ 55us) write_op.cc:312] APPLY: finished
0504 20:01:27.196738 (+ 279us) log.cc:844] Serialized 65 byte log entry
0504 20:01:27.284180 (+ 87442us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:27.302573 (+ 18393us) write_op.cc:454] Released schema lock
0504 20:01:27.314793 (+ 12220us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:27.440748 (+125955us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:27.442456 (+ 1708us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":699426,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":565400,"apply.run_cpu_time_us":14413,"apply.run_wall_time_us":935605,"num_ops":8,"prepare.queue_time_us":198902,"prepare.run_cpu_time_us":32007,"prepare.run_wall_time_us":1122263,"raft.queue_time_us":817811,"raft.run_cpu_time_us":47655,"raft.run_wall_time_us":1280462,"replication_time_us":3282169,"thread_start_us":2014127,"threads_started":5,"wal-append.queue_time_us":689127,"wal-append.run_cpu_time_us":79778,"wal-append.run_wall_time_us":4263849}]]}
I20260504 20:01:29.897956 15179 consensus_queue.cc:1048] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:29.878070 15141 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.879684 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.882148 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.882496 15094 consensus_queue.cc:1059] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:29.884588 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:29.885393 14952 consensus_queue.cc:1059] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:29.886283 15210 raft_consensus.cc:697] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.890897 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.903270 12894 raft_consensus.cc:2393] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:29.903666 15210 consensus_queue.cc:237] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.828238 13499 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:29.894122 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:29.900003 15141 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:29.908816 12747 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.911310 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:29.914592 12714 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
W20260504 20:01:29.918198 12716 leader_election.cc:336] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.372s (SENT)
I20260504 20:01:29.919341 15140 consensus_queue.cc:1059] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260504 20:01:29.920586 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:29.921020 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.921751 12895 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:29.869711 15167 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:29.930248 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:29.930428 15136 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:29.937495 15201 consensus_queue.cc:1059] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:29.920621 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.938869 12892 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:29.939280 15090 raft_consensus.cc:3055] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:29.939724 15090 raft_consensus.cc:740] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.939901 12820 raft_consensus.cc:1275] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:29.940016 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:29.940754 15090 consensus_queue.cc:260] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.942550 12820 raft_consensus.cc:1275] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:29.942484 15201 consensus_queue.cc:1059] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:29.943523 15201 raft_consensus.cc:3055] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:29.943580 15208 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.944718 15090 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:29.945688 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:29.930395 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 102263us
I20260504 20:01:29.944104 15201 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.952732 12821 raft_consensus.cc:1240] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260504 20:01:29.940824 12818 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:29.963411 12747 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:29.920699 15140 raft_consensus.cc:3055] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:29.965518 15140 raft_consensus.cc:740] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:29.940771 15116 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.967907 12819 raft_consensus.cc:1275] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:29.968400 12742 raft_consensus.cc:2368] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6.
I20260504 20:01:29.968348 15140 consensus_queue.cc:260] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:29.893309 15188 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:29.959796 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:29.974699 13499 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:29.976452 13499 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:29.981428 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.982676 12892 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:29.990401 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.990904 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.993276 12818 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260504 20:01:29.969946 15140 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:29.998286 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:30.000882 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:30.001294 12821 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:29.998829 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:29.970036 15208 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:30.005137 12894 raft_consensus.cc:1240] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:30.008973 12820 raft_consensus.cc:2368] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
I20260504 20:01:30.010164 12863 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:30.014776 14952 consensus_queue.cc:1059] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:30.015940 15216 consensus_queue.cc:1059] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260504 20:01:30.016234 15246 raft_consensus.cc:3055] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:30.016367 14792 consensus_queue.cc:1059] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:30.016820 15246 raft_consensus.cc:740] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:30.015964 15116 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:30.017796 15246 consensus_queue.cc:260] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.022346 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.022895 15246 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:30.023304 12893 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:30.026033 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.026212 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.026852 12890 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:30.018478 15027 raft_consensus.cc:3055] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:30.027727 15027 raft_consensus.cc:740] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:30.028746 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.028689 15027 consensus_queue.cc:260] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:30.011370 15009 consensus_queue.cc:1048] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260504 20:01:30.029646 12893 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:30.018793 12824 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:30.030145 15027 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:30.019163 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:29.998097 15082 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:30.018785 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.018344 15216 consensus_queue.cc:1059] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260504 20:01:30.032169 15009 consensus_queue.cc:1048] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:01:30.017825 15116 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:30.030759 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.033778 15116 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:30.034236 15116 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:30.034339 14899 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:30.035471 14823 raft_consensus.cc:3055] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:30.031899 15141 consensus_queue.cc:1059] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260504 20:01:30.012230 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:30.035815 15164 raft_consensus.cc:3055] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:30.040915 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.034409 12890 raft_consensus.cc:2410] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260504 20:01:30.035874 14823 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:30.001926 15179 consensus_queue.cc:1048] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260504 20:01:30.048170 12898 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:29.995201 15136 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:30.059407 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.071130 12747 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:30.047793 15106 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:30.038019 15188 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:30.075317 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.075426 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:30.076399 12749 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:30.080570 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:30.080691 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:30.048635 14979 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:30.049619 15164 raft_consensus.cc:740] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:30.085908 12744 raft_consensus.cc:2393] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:30.087411 12742 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:30.087734 12863 leader_election.cc:400] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
I20260504 20:01:30.092650 12863 leader_election.cc:403] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260504 20:01:30.092703 12892 raft_consensus.cc:1240] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260504 20:01:30.080862 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.091867 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:30.088912 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 174388us
I20260504 20:01:30.094651 12745 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:30.074158 15106 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:30.103892 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.105552 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.107612 15106 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:30.038573 15244 consensus_queue.cc:1059] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
W20260504 20:01:30.108361 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 57822.5us
I20260504 20:01:30.038388 15201 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.132167 12714 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.130261 12747 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:30.133152 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 117660us
W20260504 20:01:30.133607 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 170562us
I20260504 20:01:30.138039 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.126549 12898 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:30.143833 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.138231 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.147761 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.322520 12747 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:30.323741 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:30.324918 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.149874 12821 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:30.326167 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:30.326404 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.326977 12750 raft_consensus.cc:2393] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:30.315435 15246 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:30.119462 15210 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:30.328275 15246 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:30.331355 15246 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:30.120208 15094 raft_consensus.cc:3055] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:30.115546 15188 consensus_queue.cc:1059] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260504 20:01:30.181641 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:30.127418 15161 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.126195 14823 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.127271 15164 consensus_queue.cc:260] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:30.221478 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 89418.7us
I20260504 20:01:30.223609 12820 raft_consensus.cc:1275] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:30.223745 15208 raft_consensus.cc:3055] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
W20260504 20:01:30.226395 12633 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 0e8eba4dbb8d436f81062ca3bbf46788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 5ff0ef72ec3b4667893a0420d3c7f923 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:30.347949 15208 raft_consensus.cc:740] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:30.348196 12633 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 174670us
I20260504 20:01:30.232463 15023 raft_consensus.cc:3055] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:30.348968 15208 consensus_queue.cc:260] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.349591 15023 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:30.350706 15023 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:30.351955 15023 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:30.233176 12816 raft_consensus.cc:1240] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260504 20:01:30.233986 14952 raft_consensus.cc:3055] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Stepping down as leader of term 4
W20260504 20:01:30.235167 15245 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260504 20:01:30.353466 14952 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:30.233495 15248 raft_consensus.cc:3055] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:30.354688 15248 raft_consensus.cc:740] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:30.354611 14952 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.355798 12714 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.356163 14952 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:30.243480 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.358530 15188 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:30.359076 15188 raft_consensus.cc:697] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:30.359423 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.244850 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.256304 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 113732us
I20260504 20:01:30.360105 15188 consensus_queue.cc:237] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.258184 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.360134 12893 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:30.361366 12896 raft_consensus.cc:2393] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:30.257190 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.222256 15201 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:30.262518 12742 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:30.354640 15164 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:30.362756 12791 connection.cc:254] RPC call timeout handler was delayed by -1.5942235500000379s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2821s for a response
I20260504 20:01:30.363245 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:30.363466 12791 connection.cc:254] RPC call timeout handler was delayed by -1.0408687359999931s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
I20260504 20:01:30.364164 15198 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:30.364696 12742 raft_consensus.cc:1240] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:30.364728 15198 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.263622 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.366304 12899 raft_consensus.cc:2410] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:30.366348 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:30.367084 15198 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:30.367547 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:30.367663 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.286723 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.302068 13529 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260504 20:01:30.369057 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:30.371251 12816 raft_consensus.cc:2393] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:30.312881 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.322296 15082 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:30.177340 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.327631 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.373229 12899 raft_consensus.cc:2393] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:01:30.338469 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.339064 12895 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:30.384471 12714 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.335705 12747 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
W20260504 20:01:30.340085 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.340941 15094 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:30.343432 12894 raft_consensus.cc:2376] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4. Re-sending same reply.
I20260504 20:01:30.346237 12741 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260504 20:01:30.346801 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:30.347007 15161 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:30.392285 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.347644 12743 raft_consensus.cc:2368] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6.
I20260504 20:01:30.353052 14823 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:30.393735 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:30.360590 12897 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:30.361979 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.259609 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 105845us
I20260504 20:01:30.368952 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:30.396661 12821 raft_consensus.cc:2393] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:30.369059 12749 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:30.306551 15116 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:30.372090 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.403595 12819 raft_consensus.cc:1275] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:30.404085 12745 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:30.387712 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.388496 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.342034 15210 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
W20260504 20:01:30.403748 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.406320 12894 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:30.391062 12863 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:30.412273 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.423553 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.424597 12744 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:30.411422 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.425695 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.426244 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:30.426661 12894 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:30.427134 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:30.430270 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.430581 12821 raft_consensus.cc:1240] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:30.431267 12897 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:30.425381 12893 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:30.433881 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.434508 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.434723 12750 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260504 20:01:30.435513 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:30.438899 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.435772 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:30.440105 12899 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 6
I20260504 20:01:30.442323 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:30.443555 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 59264.4us
I20260504 20:01:30.444487 12714 leader_election.cc:400] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:01:30.447417 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.447557 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.448159 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.451916 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.452145 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.452385 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:30.453143 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.454110 13497 leader_election.cc:336] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.501s (SENT)
I20260504 20:01:30.454479 15116 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:30.454686 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.455636 15023 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:30.441648 15092 consensus_queue.cc:1048] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260504 20:01:30.456722 12823 raft_consensus.cc:1240] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:30.458324 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.459599 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.460175 12898 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:30.411181 15249 raft_consensus.cc:2804] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:30.461082 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.461305 15249 raft_consensus.cc:697] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:30.465245 12895 raft_consensus.cc:2393] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:30.423869 15179 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:30.448989 12893 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
W20260504 20:01:30.475436 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 85329.5us
W20260504 20:01:30.476717 12716 leader_election.cc:336] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.096s (SENT)
W20260504 20:01:30.477443 12716 leader_election.cc:336] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.108s (SENT)
I20260504 20:01:30.481844 12741 raft_consensus.cc:1275] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:30.483134 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.483963 12893 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:30.485963 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.486328 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.486816 12743 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:30.483717 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:30.481501 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.355841 15248 consensus_queue.cc:260] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.489881 12716 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.435087 15251 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:30.491856 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.498924 12895 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:30.499387 12823 raft_consensus.cc:1275] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:30.447058 15208 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:30.446789 14979 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260504 20:01:30.467927 15210 consensus_queue.cc:1048] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260504 20:01:30.463353 15094 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.462493 15249 consensus_queue.cc:237] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:30.556880 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.500088 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:30.756953 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.757738 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.525530 15009 consensus_queue.cc:1048] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.235s
I20260504 20:01:30.760854 12897 raft_consensus.cc:2410] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260504 20:01:30.604966 15179 raft_consensus.cc:697] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:30.765282 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:30.767300 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 289868us
W20260504 20:01:30.768280 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 323860us
I20260504 20:01:30.649861 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.770404 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.771016 12749 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:30.771816 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.770601 12714 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.773438 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:30.774221 13530 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:30.776149 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:30.484105 15106 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.779208 12891 raft_consensus.cc:1240] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:30.503500 12818 raft_consensus.cc:1275] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260504 20:01:30.600903 13531 raft_consensus.cc:1240] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:30.525965 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.662251 12747 raft_consensus.cc:1240] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:30.783939 12898 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:30.781481 15106 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:30.506771 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:30.794312 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.795945 12898 raft_consensus.cc:2410] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260504 20:01:30.796096 12899 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
W20260504 20:01:30.785938 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.787914 12749 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:30.783169 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:30.660293 12741 raft_consensus.cc:1275] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:30.783229 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.810992 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.813983 12749 raft_consensus.cc:1275] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:30.816078 12741 raft_consensus.cc:1275] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260504 20:01:30.821532 12899 raft_consensus.cc:1275] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:30.817102 12741 pending_rounds.cc:85] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting all ops after (but not including) 2
I20260504 20:01:30.819677 12742 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:30.832267 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:30.595037 15258 log.cc:927] Time spent T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20260504 20:01:30.830521 12741 pending_rounds.cc:107] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260504 20:01:30.765730 15179 consensus_queue.cc:237] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:30.793123 15248 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:30.838120 12741 raft_consensus.cc:2889] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:30.839996 12897 raft_consensus.cc:1240] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:30.835770 12749 raft_consensus.cc:1240] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:30.841257 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 70992us
I20260504 20:01:30.835181 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:30.855693 12823 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:30.482961 15092 consensus_queue.cc:1059] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.349s
I20260504 20:01:30.868024 13525 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:30.868856 12895 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260504 20:01:30.594731 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:30.871665 12895 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:30.857205 12899 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:30.827364 15009 consensus_queue.cc:1048] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260504 20:01:30.875517 12743 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:30.832722 15094 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:30.869367 15274 consensus_queue.cc:1059] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260504 20:01:30.882294 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.884943 12714 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:30.890517 12750 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
W20260504 20:01:30.893620 15263 log.cc:927] Time spent T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.454s user 0.001s sys 0.000s
W20260504 20:01:30.894110 15201 consensus_meta.cc:287] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.531s user 0.009s sys 0.002s
I20260504 20:01:30.897111 12892 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:30.894273 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.895557 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.878285 15116 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:30.900472 12823 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:30.901914 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:30.883858 15164 consensus_meta.cc:287] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.521s user 0.011s sys 0.003s
I20260504 20:01:30.910923 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.912385 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.912470 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.916086 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 357286us
I20260504 20:01:30.914556 12895 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:30.915071 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.918691 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:30.919430 12716 leader_election.cc:336] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.507s (SENT)
I20260504 20:01:30.920091 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.924233 13497 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.926919 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.927541 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:30.928237 12893 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:30.927662 12899 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:30.937887 15023 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:30.938827 13530 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260504 20:01:30.960397 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:30.982690 13497 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:30.983786 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:30.985986 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 171038us
I20260504 20:01:30.986264 12899 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:01:30.983822 12745 consensus_meta.cc:287] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.560s user 0.012s sys 0.000s
I20260504 20:01:30.987900 12745 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:30.988608 12746 raft_consensus.cc:1240] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:30.990804 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:30.994120 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 167836us
I20260504 20:01:30.991402 14952 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:31.005898 15023 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:31.006901 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.008750 12863 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:30.975791 15276 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:30.990362 15246 raft_consensus.cc:3055] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
W20260504 20:01:31.022054 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 87115.1us
I20260504 20:01:31.022979 13497 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:31.022980 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.024667 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.014791 15246 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:31.024849 12893 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:31.025758 12899 raft_consensus.cc:2393] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:31.028528 13497 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:31.032219 15164 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:31.032739 12818 raft_consensus.cc:1240] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:31.040055 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:31.038640 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 224377us
I20260504 20:01:31.041249 12893 raft_consensus.cc:2393] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:31.021742 15077 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:31.042898 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 140594us
W20260504 20:01:31.044592 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:31.047245 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:31.048936 12817 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.711s user 0.002s sys 0.018s
W20260504 20:01:31.049320 12817 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.711s user 0.002s sys 0.018s
I20260504 20:01:31.038583 15249 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:31.034901 15262 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:31.060868 13532 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:31.062026 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:31.072408 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 146696us
W20260504 20:01:31.078366 12716 leader_election.cc:336] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.551s (SENT)
I20260504 20:01:31.018061 15276 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.065634 12864 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:31.061858 15249 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:31.080760 12714 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:31.082406 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:31.067243 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.097373 15287 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:31.065623 15179 consensus_queue.cc:1059] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260504 20:01:31.098910 15287 consensus_queue.cc:1059] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:01:31.099855 15287 raft_consensus.cc:3055] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:31.079749 15262 raft_consensus.cc:697] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:31.100239 15287 raft_consensus.cc:740] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:31.100952 15262 consensus_queue.cc:237] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:31.101150 15287 consensus_queue.cc:260] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:31.102504 15287 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:31.104630 14823 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.701s user 0.013s sys 0.000s
I20260504 20:01:31.104811 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.105718 12890 raft_consensus.cc:2410] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:01:31.106362 15092 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:31.106137 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.107075 15092 raft_consensus.cc:697] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:31.107426 15253 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:31.108059 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.109651 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.111533 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:31.112099 15276 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:31.106220 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.113077 12899 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:31.107240 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.117506 12791 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:31.096282 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 100594us
I20260504 20:01:31.118794 12715 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:31.124569 12893 raft_consensus.cc:2393] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:31.126394 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 45946.5us
W20260504 20:01:31.158957 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.055316 15077 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.114372 15249 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:31.172399 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 114142us
W20260504 20:01:31.182404 15278 log.cc:927] Time spent T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260504 20:01:31.165261 15281 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:31.194712 12897 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:31.146752 14979 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:31.298942 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 94396.5us
W20260504 20:01:31.315037 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.124230 15198 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260504 20:01:31.201083 15290 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:31.334955 15291 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:31.336033 12893 raft_consensus.cc:1240] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
W20260504 20:01:31.342664 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 225231us
I20260504 20:01:31.438752 12791 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:31.091847 15246 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.441499 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.196689 15119 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:31.352087 15094 consensus_queue.cc:1059] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260504 20:01:31.365541 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.444458 15253 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:31.449719 15253 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.451354 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1804) took 1207 ms. Trace:
I20260504 20:01:31.452124 15253 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:31.452219 12891 rpcz_store.cc:276] 0504 20:01:30.243696 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.264089 (+ 20393us) service_pool.cc:224] Handling call
0504 20:01:30.284836 (+ 20747us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:30.779822 (+494986us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:31.451081 (+671259us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b5400207bd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:31.451279 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1136221568}
I20260504 20:01:31.425652 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:31.453974 15284 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:31.454216 13530 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:31.454571 15284 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.455529 14979 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:31.455412 15119 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:31.456063 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.456180 14979 raft_consensus.cc:697] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:31.456910 15284 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:31.440824 15256 log.cc:927] Time spent T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.240s user 0.000s sys 0.001s
I20260504 20:01:31.201076 15281 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:31.332372 15164 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:31.108209 15092 consensus_queue.cc:237] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:31.483958 15248 consensus_meta.cc:287] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.623s user 0.011s sys 0.000s
W20260504 20:01:31.347640 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.445246 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:31.458006 12890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1810) took 1195 ms. Trace:
I20260504 20:01:31.460858 12790 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:31.487550 12890 rpcz_store.cc:276] 0504 20:01:30.262934 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.323437 (+ 60503us) service_pool.cc:224] Handling call
0504 20:01:31.457745 (+1134308us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b5400002b54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:31.457921 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1796009216}
W20260504 20:01:31.439020 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 360682us
W20260504 20:01:31.488956 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.457875 15119 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:31.472914 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:31.473320 15282 log.cc:927] Time spent T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20260504 20:01:31.472537 15283 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:31.476243 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:31.477144 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 311345us
I20260504 20:01:31.478813 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:31.440394 15291 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:31.482549 13531 raft_consensus.cc:1240] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:31.480993 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:31.507157 12893 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:31.471217 15246 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:31.458614 15116 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:31.462672 15262 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:31.504665 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 43867.3us
I20260504 20:01:31.457340 14979 consensus_queue.cc:237] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:31.506150 12714 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:31.507944 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.502939 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:31.509266 15291 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:31.509905 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:31.510228 15210 raft_consensus.cc:3055] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:31.538235 15210 raft_consensus.cc:740] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:31.539440 15210 consensus_queue.cc:260] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.540669 15210 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:31.509711 15283 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:31.487540 15164 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.543115 15164 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:31.479643 15281 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:31.544977 15077 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:31.545682 15116 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.546531 13532 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40940 (request call id 311) took 1057 ms. Trace:
I20260504 20:01:31.546943 13532 rpcz_store.cc:276] 0504 20:01:30.489343 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.489910 (+ 567us) service_pool.cc:224] Handling call
0504 20:01:31.546302 (+1056392us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54004b0fd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:31.546438 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":282295168}
I20260504 20:01:31.548296 15116 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:31.549846 12891 raft_consensus.cc:2393] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:01:31.514315 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 69193.5us
I20260504 20:01:31.552816 12791 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:31.527971 15262 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:31.557754 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 51701.6us
I20260504 20:01:31.558413 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.559082 12891 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
W20260504 20:01:31.562266 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:31.529362 13529 consensus_meta.cc:287] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.659s user 0.010s sys 0.002s
I20260504 20:01:31.530602 15292 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:31.530514 15296 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260504 20:01:31.543015 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:31.522429 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.579013 12895 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 6
I20260504 20:01:31.567438 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:31.555987 12748 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:31.569118 12791 leader_election.cc:400] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6.
I20260504 20:01:31.582849 12791 leader_election.cc:403] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260504 20:01:31.585608 12864 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:31.596517 15140 raft_consensus.cc:2804] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:31.597450 15140 raft_consensus.cc:697] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:31.604178 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:31.605141 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 46631.4us
I20260504 20:01:31.605998 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:31.607947 12891 raft_consensus.cc:2410] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260504 20:01:31.543115 15283 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:31.579728 14952 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260504 20:01:31.621081 15299 log.cc:927] Time spent T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
W20260504 20:01:31.587755 15208 consensus_meta.cc:287] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.807s user 0.008s sys 0.005s
I20260504 20:01:31.581775 13529 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:31.624118 12899 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:31.614456 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:31.622165 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 53271.6us
I20260504 20:01:31.627009 12896 raft_consensus.cc:2410] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:31.585204 15292 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:31.638676 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 113537us
I20260504 20:01:31.641465 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:31.615451 15198 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:31.621282 14952 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:31.709364 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 79331us
I20260504 20:01:31.686372 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:31.692126 12890 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
W20260504 20:01:31.707998 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 51362.5us
I20260504 20:01:31.622705 15077 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6.
I20260504 20:01:31.710448 12791 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:31.708408 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 91877.8us
W20260504 20:01:31.713263 12716 leader_election.cc:336] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.689s (SENT)
I20260504 20:01:31.622052 15140 consensus_queue.cc:237] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:31.622561 15287 consensus_queue.cc:1059] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.212s
W20260504 20:01:31.787843 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 186398us
I20260504 20:01:31.884918 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.789392 12818 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:31.642030 13529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40940 (request call id 310) took 1177 ms. Trace:
I20260504 20:01:31.713989 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:31.880303 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 257351us
W20260504 20:01:31.835578 12892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1800) took 1641 ms (client timeout 2125 ms). Trace:
I20260504 20:01:31.787642 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:31.850353 15294 log.cc:927] Time spent T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
W20260504 20:01:32.127266 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.127820 12714 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:31.932363 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 222049us
I20260504 20:01:32.027619 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:31.982506 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:31.998723 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.131433 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:31.933079 12892 rpcz_store.cc:269] 0504 20:01:30.215923 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.220035 (+ 4112us) service_pool.cc:224] Handling call
0504 20:01:31.835289 (+1615254us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b54003a39d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:31.835487 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1229660416}
W20260504 20:01:32.127756 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.005491 12895 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:31.850652 15297 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20260504 20:01:31.974105 12893 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:01:31.928426 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:31.929025 12899 raft_consensus.cc:2410] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260504 20:01:31.929993 12891 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:01:32.070518 15309 log.cc:927] Time spent T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
W20260504 20:01:32.127064 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.156531 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.128243 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1629) took 1366 ms. Trace:
W20260504 20:01:32.049785 15291 consensus_meta.cc:287] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.515s user 0.007s sys 0.005s
I20260504 20:01:31.852162 15305 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:31.880249 15300 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:32.131927 15198 raft_consensus.cc:697] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:31.852478 15303 consensus_queue.cc:1048] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.295s
I20260504 20:01:32.159549 15291 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.168605 15305 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.137792 15116 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:32.171149 15305 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:32.171258 15291 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:32.139413 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 426174us
I20260504 20:01:32.126446 12822 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:32.172039 15287 consensus_queue.cc:1059] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:32.141757 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:31.715677 14952 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:32.173353 15287 raft_consensus.cc:3055] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:32.173840 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:31.998698 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.146241 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:32.147478 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:32.130904 15090 raft_consensus.cc:3055] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:32.148557 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:31.954314 15296 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260504 20:01:32.151819 15289 log.cc:927] Time spent T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.950s user 0.008s sys 0.009s
W20260504 20:01:32.154263 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.115062 15249 consensus_meta.cc:287] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.875s user 0.010s sys 0.002s
I20260504 20:01:32.178922 15090 raft_consensus.cc:740] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:32.188167 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:32.188055 15198 consensus_queue.cc:237] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.189961 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1801) took 1394 ms. Trace:
I20260504 20:01:32.190392 12825 rpcz_store.cc:276] 0504 20:01:30.795344 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.796214 (+ 870us) service_pool.cc:224] Handling call
0504 20:01:30.835400 (+ 39186us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:30.858930 (+ 23530us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:30.858983 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:30.883686 (+ 24703us) log.cc:844] Serialized 26 byte log entry
0504 20:01:31.029052 (+145366us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:31.029123 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:31.029415 (+ 292us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:32.151000 (+1121585us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:32.189533 (+ 38533us) raft_consensus.cc:1685] finished
0504 20:01:32.189571 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:32.189839 (+ 268us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10542,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":117,"spinlock_wait_cycles":23985920,"thread_start_us":114809,"threads_started":1,"wal-append.queue_time_us":185427}
I20260504 20:01:32.192543 15249 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
W20260504 20:01:32.194392 15316 log.cc:927] Time spent T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260504 20:01:32.194763 15249 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:32.188189 15090 consensus_queue.cc:260] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.197603 15090 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:32.198935 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:32.199646 13527 raft_consensus.cc:2376] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3. Re-sending same reply.
I20260504 20:01:32.202632 15262 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:32.203197 15210 consensus_meta.cc:287] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.662s user 0.010s sys 0.003s
W20260504 20:01:32.204802 12748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1609) took 1882 ms (client timeout 1968 ms). Trace:
I20260504 20:01:32.184010 15296 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:32.206295 15296 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:31.933899 13529 rpcz_store.cc:276] 0504 20:01:30.464436 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.474823 (+ 10387us) service_pool.cc:224] Handling call
0504 20:01:31.638538 (+1163715us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:32.205291 12748 rpcz_store.cc:269] 0504 20:01:30.342936 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.343108 (+ 172us) service_pool.cc:224] Handling call
0504 20:01:32.204579 (+1861471us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400242f54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:32.204722 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2479793024}
W20260504 20:01:32.211963 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.156332 12791 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:32.135761 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1668) took 1030 ms. Trace:
I20260504 20:01:32.214169 12895 rpcz_store.cc:276] 0504 20:01:31.105688 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:31.118294 (+ 12606us) service_pool.cc:224] Handling call
0504 20:01:31.542895 (+424601us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d3adc 00007f01598d3a46 000000000035a3c9 00007f016190b8e0 00007f01619d4784 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:32.135689 (+592794us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":678230528}
I20260504 20:01:32.215682 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.216467 12895 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:32.157155 12746 rpcz_store.cc:276] 0504 20:01:30.761866 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.823283 (+ 61417us) service_pool.cc:224] Handling call
0504 20:01:30.988430 (+165147us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:30.988976 (+ 546us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:32.128009 (+1139033us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54002a0650. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:32.128167 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338200832}
I20260504 20:01:31.696727 15292 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:32.161008 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:32.220541 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.165369 12714 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:32.222007 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 56813.1us
W20260504 20:01:32.166688 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.141305 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.172557 12716 leader_election.cc:336] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.774s (SENT)
I20260504 20:01:32.173905 15287 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:32.145936 15164 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:32.162189 15188 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s user 0.004s sys 0.013s
I20260504 20:01:32.219143 13499 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:31.927716 15307 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:32.184356 15246 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.660s user 0.011s sys 0.004s
I20260504 20:01:32.228672 12864 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:32.230070 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 73919.7us
W20260504 20:01:32.231361 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 58842.9us
I20260504 20:01:32.232676 12863 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:32.234825 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1805) took 1246 ms. Trace:
I20260504 20:01:32.236444 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:32.235774 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:32.178859 15300 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.226208 12714 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:32.244953 12890 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:01:32.261868 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.234508 12716 leader_election.cc:336] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.949s (SENT)
I20260504 20:01:32.241760 12891 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:32.225222 15025 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:32.251039 15317 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:32.324523 15295 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:32.262117 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:32.323560 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 97449.5us
I20260504 20:01:32.260660 15141 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:32.228071 15249 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260504 20:01:32.383889 15281 consensus_meta.cc:287] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.837s user 0.016s sys 0.000s
I20260504 20:01:32.402424 15249 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:32.235422 12824 rpcz_store.cc:276] 0504 20:01:30.988762 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.988923 (+ 161us) service_pool.cc:224] Handling call
0504 20:01:31.005457 (+ 16534us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:31.016249 (+ 10792us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:31.016285 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:31.022453 (+ 6168us) log.cc:844] Serialized 26 byte log entry
0504 20:01:31.139063 (+116610us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:31.139186 (+ 123us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:31.139382 (+ 196us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:32.234520 (+1095138us) raft_consensus.cc:1685] finished
0504 20:01:32.234561 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:32.234742 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5212,"raft.queue_time_us":12413,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":5430,"spinlock_wait_cycles":7040,"thread_start_us":94935,"threads_started":1,"wal-append.queue_time_us":581615}
I20260504 20:01:32.323295 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.403030 15281 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.404201 12895 raft_consensus.cc:2410] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:01:32.405253 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.405388 15281 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:32.336642 15246 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:32.382504 15025 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:32.406571 12896 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1806) took 2162 ms (client timeout 2108 ms). Trace:
I20260504 20:01:32.403041 15249 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:32.407743 12898 raft_consensus.cc:1240] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 3. Ops: []
I20260504 20:01:32.408149 12895 raft_consensus.cc:2376] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5. Re-sending same reply.
I20260504 20:01:32.409219 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.365970 12899 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:32.410538 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.411681 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.412511 12899 raft_consensus.cc:2410] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:32.413355 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.366475 14899 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:32.415015 14952 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:32.366485 12894 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1824) took 2009 ms (client timeout 2126 ms). Trace:
I20260504 20:01:32.359922 15287 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:32.416109 12894 rpcz_store.cc:269] 0504 20:01:30.357018 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.436809 (+ 79791us) service_pool.cc:224] Handling call
0504 20:01:32.366218 (+1929409us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b5400208ad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:32.366411 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1395672448}
I20260504 20:01:32.416589 15287 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:32.320230 15307 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.338698 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:32.230562 15164 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.369647 12716 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:32.371266 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.452039 12891 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:32.452600 15164 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:32.454095 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.456095 12747 raft_consensus.cc:3055] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:32.375685 12893 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
W20260504 20:01:32.383021 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 147365us
I20260504 20:01:32.457731 12791 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:32.457934 15322 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:32.458221 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:32.458683 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 224220us
I20260504 20:01:32.459071 12744 raft_consensus.cc:2393] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:32.459261 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:32.383076 12890 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:32.384932 12714 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:32.460320 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 75438.9us
I20260504 20:01:32.460994 12714 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:32.406317 15246 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.337702 15251 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:32.468557 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:32.469074 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:32.469489 12897 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:32.470242 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1669) took 1327 ms. Trace:
I20260504 20:01:32.417680 12894 raft_consensus.cc:1275] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:32.278741 12892 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:32.418495 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:32.409564 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1634) took 2049 ms. Trace:
I20260504 20:01:32.472430 12748 raft_consensus.cc:2393] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:32.472345 12898 rpcz_store.cc:276] 0504 20:01:30.360636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.826997 (+466361us) service_pool.cc:224] Handling call
0504 20:01:32.407544 (+1580547us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:32.408234 (+ 690us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:32.409342 (+ 1108us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54003b9550. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:32.409499 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3627776384}
I20260504 20:01:32.419034 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:32.378501 15317 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:32.417230 14952 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:32.407052 12896 rpcz_store.cc:269] 0504 20:01:30.244468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.403512 (+159044us) service_pool.cc:224] Handling call
0504 20:01:32.406267 (+2002755us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54003b9554. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:32.406473 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2739946880}
I20260504 20:01:32.478298 12750 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:32.362491 15319 raft_consensus.cc:2804] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:32.442665 12899 raft_consensus.cc:2376] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5. Re-sending same reply.
I20260504 20:01:32.390765 15304 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:32.450596 15313 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20260504 20:01:32.450402 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:32.456473 12747 raft_consensus.cc:740] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:32.468338 14979 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:32.408193 15025 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:32.486614 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:32.487576 12822 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:32.488310 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:32.489065 12816 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:32.491120 12865 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.467056 12716 leader_election.cc:336] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.032s (SENT)
I20260504 20:01:32.464282 12749 raft_consensus.cc:1275] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:32.461884 12791 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:32.470672 12897 rpcz_store.cc:276] 0504 20:01:31.142994 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:31.199493 (+ 56499us) service_pool.cc:224] Handling call
0504 20:01:32.468330 (+1268837us) spinlock_profiling.cc:243] Waited 99 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:32.470166 (+ 1836us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227722112}
I20260504 20:01:32.439731 12741 raft_consensus.cc:1275] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:32.468478 15077 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:32.494419 12744 raft_consensus.cc:1275] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260504 20:01:32.494853 12744 pending_rounds.cc:85] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting all ops after (but not including) 1
I20260504 20:01:32.495091 12744 pending_rounds.cc:107] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260504 20:01:32.495332 12744 raft_consensus.cc:2889] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:32.480589 12747 consensus_queue.cc:260] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.496179 12747 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:32.496392 12821 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8. Ops: []
I20260504 20:01:32.496807 12744 raft_consensus.cc:3055] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:32.497156 12744 raft_consensus.cc:740] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:32.497644 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:32.498924 15300 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:32.499794 12742 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:32.499240 12821 raft_consensus.cc:1240] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:32.502111 12821 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260504 20:01:32.495198 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:32.502195 12741 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:32.502205 12746 raft_consensus.cc:2368] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
I20260504 20:01:32.507597 12821 raft_consensus.cc:1275] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:32.508589 12750 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:32.510618 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:32.510910 12865 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.511518 12741 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:32.512288 12865 connection.cc:254] RPC call timeout handler was delayed by -1.5406679189999295s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
W20260504 20:01:32.512713 12865 connection.cc:254] RPC call timeout handler was delayed by -1.4668679189999239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1585s for a response
I20260504 20:01:32.512951 12823 raft_consensus.cc:2393] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:32.513150 12865 connection.cc:254] RPC call timeout handler was delayed by -1.4128679189999502s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
W20260504 20:01:32.513630 12865 connection.cc:254] RPC call timeout handler was delayed by -1.3372679189999417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260504 20:01:32.514070 12865 connection.cc:254] RPC call timeout handler was delayed by -1.3327679189999344s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1734s for a response
W20260504 20:01:32.514212 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 47193.6us
W20260504 20:01:32.515622 12716 leader_election.cc:336] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.126s (SENT)
W20260504 20:01:32.516832 12716 leader_election.cc:336] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.147s (SENT)
I20260504 20:01:32.517367 15327 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:32.518204 12716 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:32.518378 15327 raft_consensus.cc:697] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:32.514547 12865 connection.cc:254] RPC call timeout handler was delayed by -1.3237679189999199s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1744s for a response
W20260504 20:01:32.519088 12865 connection.cc:254] RPC call timeout handler was delayed by -1.3120679189999009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1757s for a response
W20260504 20:01:32.519484 12865 connection.cc:254] RPC call timeout handler was delayed by -1.2697679189999462s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
W20260504 20:01:32.520027 12865 connection.cc:254] RPC call timeout handler was delayed by -1.208567918999961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18720000000000003s for a response
W20260504 20:01:32.520394 12865 connection.cc:254] RPC call timeout handler was delayed by -1.1626679190000004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19230000000000003s for a response
I20260504 20:01:32.519441 15327 consensus_queue.cc:237] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.522732 15077 raft_consensus.cc:697] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:32.517557 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:32.524051 15077 consensus_queue.cc:237] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.529683 15249 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.517912 14979 raft_consensus.cc:697] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:32.510069 15281 consensus_queue.cc:1048] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260504 20:01:32.531508 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:32.431872 15296 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.553584 12817 raft_consensus.cc:1275] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:32.532521 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:32.554836 12741 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:32.532699 15246 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:32.469266 15322 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:32.468195 15291 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:32.503123 12744 consensus_queue.cc:260] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:32.524623 12865 connection.cc:254] RPC call timeout handler was delayed by -1.122167918999935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
I20260504 20:01:32.537209 12747 raft_consensus.cc:1275] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:32.537384 12819 raft_consensus.cc:1240] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260504 20:01:32.462366 15141 raft_consensus.cc:697] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:32.541769 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:32.543020 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:32.546679 12749 raft_consensus.cc:1240] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:32.534457 15274 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:32.500682 15251 raft_consensus.cc:697] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:32.482015 15319 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:32.478489 15317 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:32.613672 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 151946us
I20260504 20:01:32.554636 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:32.574810 13531 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 383) took 1051 ms. Trace:
W20260504 20:01:32.582065 12865 connection.cc:254] RPC call timeout handler was delayed by -1.0645679189999555s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20320000000000002s for a response
W20260504 20:01:32.582479 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.584816 12743 raft_consensus.cc:1240] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:32.584842 12825 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:32.550844 12824 raft_consensus.cc:2393] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:01:32.590696 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.583645 15322 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:32.595346 15248 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:32.598274 14979 consensus_queue.cc:237] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.598500 15141 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.599346 15296 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:32.599509 15094 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260504 20:01:32.601032 15249 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:32.583933 15164 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:32.604337 12816 raft_consensus.cc:2410] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260504 20:01:32.607883 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 91071.7us
I20260504 20:01:32.613111 12744 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:32.632481 12716 leader_election.cc:336] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.169s (SENT)
I20260504 20:01:32.633069 12716 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:32.634336 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.510845 15304 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.545173 13530 raft_consensus.cc:2393] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:32.555601 15116 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:32.620715 12865 connection.cc:254] RPC call timeout handler was delayed by -1.0267679190000081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
I20260504 20:01:32.571933 15290 raft_consensus.cc:3055] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:01:32.623167 15322 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:32.622011 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.659720 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.581223 15291 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:32.557608 15303 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:32.624370 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.612608 15307 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:32.627835 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:32.621042 13531 rpcz_store.cc:276] 0504 20:01:31.523901 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:31.549276 (+ 25375us) service_pool.cc:224] Handling call
0504 20:01:31.549694 (+ 418us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:31.550177 (+ 483us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:31.550208 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:31.550811 (+ 603us) log.cc:844] Serialized 26 byte log entry
0504 20:01:31.875026 (+324215us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:31.875109 (+ 83us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:31.875224 (+ 115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:32.574560 (+699336us) raft_consensus.cc:1685] finished
0504 20:01:32.574584 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:32.574742 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1336,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":21598,"thread_start_us":323719,"threads_started":1,"wal-append.queue_time_us":633797}
I20260504 20:01:32.615713 12791 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:32.614029 15319 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:32.650868 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.653120 12716 leader_election.cc:336] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.196s (SENT)
I20260504 20:01:32.662276 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:32.664296 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.650931 15023 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:32.658810 15248 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:32.671156 13528 raft_consensus.cc:2393] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:32.669019 12747 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
W20260504 20:01:32.670363 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 54775.4us
I20260504 20:01:32.661410 15025 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:32.659133 15290 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:32.679075 12746 raft_consensus.cc:1240] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:32.680240 12791 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:32.668820 12716 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:32.615060 15274 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } attrs { replace: true } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.664773 15291 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:32.681399 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:32.684561 12791 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:32.686775 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:32.687539 12816 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:32.669147 15249 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:32.651566 15164 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.674088 15198 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:32.658977 15116 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.688200 15296 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:32.702734 15296 raft_consensus.cc:697] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:32.704207 15116 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:32.695643 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:32.685874 15025 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.705116 12817 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:32.679066 15023 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.706959 15304 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:32.708623 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.708956 15023 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:32.712244 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.712097 15023 consensus_queue.cc:1059] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:01:32.695840 15210 raft_consensus.cc:2764] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260504 20:01:32.703815 15296 consensus_queue.cc:237] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.709237 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:32.715520 12746 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:32.717049 12746 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260504 20:01:32.710258 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.713893 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:32.709784 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 52342.2us
I20260504 20:01:32.689551 15307 consensus_queue.cc:1059] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.012s
I20260504 20:01:32.718787 15025 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:32.719405 12865 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:32.720703 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:32.721477 15210 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:32.721911 15210 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:32.722283 15210 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:32.698141 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:32.715628 15164 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:32.723170 12749 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:32.723680 15164 raft_consensus.cc:3055] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:32.724083 15164 raft_consensus.cc:740] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
W20260504 20:01:32.724426 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.675189 15303 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.726109 15025 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:01:32.726742 15303 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:32.727439 15295 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:32.728808 12865 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:32.730947 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:32.731571 15304 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 20:01:32.695351 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.742151 12817 raft_consensus.cc:2410] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:01:32.753047 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:32.753738 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:32.756520 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.776515 12744 raft_consensus.cc:1275] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260504 20:01:32.777160 12744 pending_rounds.cc:85] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting all ops after (but not including) 2
I20260504 20:01:32.777486 12744 pending_rounds.cc:107] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260504 20:01:32.778115 12744 raft_consensus.cc:2889] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:32.780176 12714 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:32.792359 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 53087.2us
I20260504 20:01:32.776679 15116 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:32.793727 12865 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:32.798722 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 103138us
W20260504 20:01:32.800747 13497 leader_election.cc:336] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.629s (SENT)
I20260504 20:01:32.801218 13497 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:32.802824 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:32.803365 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:32.692027 15327 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 20:01:32.809401 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:32.723354 15274 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:32.750880 15025 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:32.915089 15336 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:01:32.915333 15332 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:32.912931 15302 consensus_queue.cc:1048] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260504 20:01:32.919267 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:32.887012 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.923230 15141 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:32.925309 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:32.775954 14899 consensus_queue.cc:1048] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.137s
I20260504 20:01:32.777453 15246 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:32.953027 15246 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.912915 15116 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:32.955130 15246 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:32.926000 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 137130us
I20260504 20:01:32.958392 12716 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:32.953649 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 406) took 1335 ms. Trace:
W20260504 20:01:32.934710 15292 consensus_meta.cc:287] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.704s user 0.014s sys 0.010s
I20260504 20:01:32.725692 15164 consensus_queue.cc:260] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:32.913846 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 113127us
I20260504 20:01:32.929526 15327 raft_consensus.cc:697] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:32.694334 15249 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:32.935943 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 142383us
I20260504 20:01:32.776225 15281 consensus_queue.cc:1048] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.154s
I20260504 20:01:32.683439 15251 consensus_queue.cc:237] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.926036 15303 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:32.947355 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:32.949832 15331 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20260504 20:01:32.940632 15322 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.955451 15116 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:32.751101 15248 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:32.960165 12895 rpcz_store.cc:276] 0504 20:01:31.618632 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:32.455093 (+836461us) service_pool.cc:224] Handling call
0504 20:01:32.460660 (+ 5567us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:32.460854 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:32.460884 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:32.460933 (+ 49us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:32.460973 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:32.461036 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:32.953440 (+492404us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b54003b9a50. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:32.953571 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12076928}
I20260504 20:01:32.930701 15287 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:32.961385 12825 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:32.966635 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:32.967545 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:32.980185 15287 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.971284 15303 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:33.014817 15303 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:33.015137 15287 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:32.982333 12716 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:33.017835 15210 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.019006 12716 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:33.020231 15188 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:33.020959 15188 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:33.021450 15188 raft_consensus.cc:697] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:33.022410 15307 consensus_queue.cc:1048] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:33.024008 15307 consensus_queue.cc:1059] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:33.024415 15210 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:33.025589 15023 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:33.026300 15023 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.027360 15291 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.028532 15023 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:32.983181 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:33.031250 15307 raft_consensus.cc:3055] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:33.031703 15307 raft_consensus.cc:740] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:33.032716 15307 consensus_queue.cc:260] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.022428 15188 consensus_queue.cc:237] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:32.987046 12865 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:32.987313 15249 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:32.988072 12750 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:32.989058 15322 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:32.989316 15179 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:32.997881 12817 raft_consensus.cc:1240] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:32.998618 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.044250 12824 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:32.913728 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:32.997426 15319 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.029510 15291 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:33.001355 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:33.003109 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:33.068771 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 81810.1us
I20260504 20:01:33.003942 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:33.005704 15290 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.009140 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:33.011401 15327 consensus_queue.cc:237] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.011912 15164 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:33.012086 12825 raft_consensus.cc:1240] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:33.013095 12819 raft_consensus.cc:1275] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260504 20:01:33.018191 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:32.983009 15116 raft_consensus.cc:2764] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260504 20:01:32.965303 15246 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:33.051528 12863 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:33.079133 12819 raft_consensus.cc:1240] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:33.052718 12716 leader_election.cc:336] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.823s (SENT)
I20260504 20:01:32.999542 15248 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:33.066448 15317 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:33.053299 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 124597us
W20260504 20:01:32.977550 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 183579us
I20260504 20:01:33.069875 12895 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:33.071839 12897 raft_consensus.cc:2393] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:33.082065 12791 connection.cc:254] RPC call timeout handler was delayed by -1.6685495559999026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260504 20:01:33.082861 12791 connection.cc:254] RPC call timeout handler was delayed by -1.6469495559999814s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
I20260504 20:01:33.057380 15322 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:33.084929 12791 connection.cc:254] RPC call timeout handler was delayed by -1.6073495559999174s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1575s for a response
I20260504 20:01:33.085076 12819 raft_consensus.cc:1240] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:33.092012 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.092453 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:33.098294 12791 connection.cc:254] RPC call timeout handler was delayed by -1.5704495559999714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
W20260504 20:01:33.098770 12791 connection.cc:254] RPC call timeout handler was delayed by -1.5236495559998957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
W20260504 20:01:33.099222 12791 connection.cc:254] RPC call timeout handler was delayed by -1.4525495559998944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
I20260504 20:01:33.099938 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:33.100854 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:33.099614 12791 connection.cc:254] RPC call timeout handler was delayed by -1.4291495559999703s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1773s for a response
I20260504 20:01:32.970675 15292 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.070529 15106 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:33.071336 15290 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:33.057699 15307 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:33.103578 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:33.102051 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:33.117626 12819 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:33.118361 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:33.105896 12816 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:33.111085 12791 connection.cc:254] RPC call timeout handler was delayed by -1.4111495559999412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
I20260504 20:01:33.103938 12714 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:33.115054 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.115878 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:33.121912 12791 connection.cc:254] RPC call timeout handler was delayed by -1.3886495559999048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
W20260504 20:01:33.122803 12791 connection.cc:254] RPC call timeout handler was delayed by -1.3859495559999004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20260504 20:01:33.122998 12648 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36606 (request call id 119) took 17058 ms (client timeout 15000 ms). Trace:
I20260504 20:01:33.123584 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:33.123176 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:33.124308 12791 connection.cc:254] RPC call timeout handler was delayed by -1.3112495559998933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
I20260504 20:01:33.124269 12750 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
W20260504 20:01:33.125128 12791 connection.cc:254] RPC call timeout handler was delayed by -1.2536495559999139s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
W20260504 20:01:33.125463 12791 connection.cc:254] RPC call timeout handler was delayed by -1.2356495559998848s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1988s for a response
I20260504 20:01:33.119976 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:33.125743 12791 connection.cc:254] RPC call timeout handler was delayed by -1.19334955599993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
W20260504 20:01:33.126081 12791 connection.cc:254] RPC call timeout handler was delayed by -1.128549555999939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
W20260504 20:01:33.126353 12791 connection.cc:254] RPC call timeout handler was delayed by -1.0934495559998823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2146s for a response
I20260504 20:01:33.121519 12824 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:33.085589 15280 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:33.094514 15319 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:33.128525 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 118367us
I20260504 20:01:33.129601 12865 leader_election.cc:400] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6.
I20260504 20:01:33.129343 12714 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:33.070093 15210 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20260504 20:01:33.134616 12865 leader_election.cc:403] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260504 20:01:33.134341 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 169240us
I20260504 20:01:33.137391 12822 raft_consensus.cc:1240] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:33.159615 12816 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:33.164124 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.105443 15322 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:33.123538 12648 rpcz_store.cc:269] 0504 20:01:16.064445 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:16.087006 (+ 22561us) service_pool.cc:224] Handling call
0504 20:01:33.122912 (+17035906us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:30.144828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:30.223114 (+ 78286us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:30.223180 (+ 66us) write_op.cc:435] Acquired schema lock
0504 20:01:30.223201 (+ 21us) tablet.cc:662] Decoding operations
0504 20:01:30.237493 (+ 14292us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:30.237615 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:30.237666 (+ 51us) tablet.cc:685] Acquiring locks for 8 operations
0504 20:01:30.238735 (+ 1069us) tablet.cc:701] Row locks acquired
0504 20:01:30.238762 (+ 27us) write_op.cc:260] PREPARE: finished
0504 20:01:30.240865 (+ 2103us) write_op.cc:270] Start()
0504 20:01:30.241057 (+ 192us) write_op.cc:276] Timestamp: P: 1777924890238916 usec, L: 0
0504 20:01:30.241100 (+ 43us) op_driver.cc:348] REPLICATION: starting
0504 20:01:30.259896 (+ 18796us) log.cc:844] Serialized 3231 byte log entry
0504 20:01:31.532658 (+1272762us) spinlock_profiling.cc:243] Waited 9.22 ms on lock 0x7b180025f300. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159910a33 00007f0159912e01 00007f0159912db9 00007f0159912d49 00007f0159912d11 00007f015991200d 0000000000379254 0000000000379099 00007f01598f3fba
0504 20:01:32.089187 (+556529us) op_driver.cc:464] REPLICATION: finished
0504 20:01:32.498153 (+408966us) write_op.cc:301] APPLY: starting
0504 20:01:32.498268 (+ 115us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:32.498531 (+ 263us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:32.498570 (+ 39us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:32.531569 (+ 32999us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:32.531934 (+ 365us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0504 20:01:32.531980 (+ 46us) write_op.cc:312] APPLY: finished
0504 20:01:32.532392 (+ 412us) log.cc:844] Serialized 65 byte log entry
0504 20:01:32.532638 (+ 246us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:32.532787 (+ 149us) write_op.cc:454] Released schema lock
0504 20:01:32.532857 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:32.533360 (+ 503us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:32.533390 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":473866,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":374527,"apply.run_cpu_time_us":3964,"apply.run_wall_time_us":35190,"num_ops":8,"prepare.queue_time_us":419179,"prepare.run_cpu_time_us":28409,"prepare.run_wall_time_us":329827,"raft.queue_time_us":449666,"raft.run_cpu_time_us":71646,"raft.run_wall_time_us":353730,"replication_time_us":1797583,"spinlock_wait_cycles":21217024,"thread_start_us":842399,"threads_started":4,"wal-append.queue_time_us":191733}]]}
I20260504 20:01:33.135481 15280 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:33.168467 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:33.169927 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 47363us
I20260504 20:01:33.165895 13499 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:33.128008 15303 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.149224 15116 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:33.175992 12791 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:33.139000 15340 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:33.178114 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 96823.7us
I20260504 20:01:33.163513 15106 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6.
I20260504 20:01:33.194037 12863 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:33.195735 12865 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:33.175556 15280 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:33.200107 12791 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:33.163686 14952 consensus_queue.cc:1059] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260504 20:01:33.198086 15251 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:33.187515 15328 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:33.212473 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:33.214107 12716 leader_election.cc:336] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.882s (SENT)
I20260504 20:01:33.192916 15246 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:33.218235 12646 catalog_manager.cc:5671] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
W20260504 20:01:33.226554 12716 leader_election.cc:336] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.009s (SENT)
I20260504 20:01:33.201531 15141 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:33.182678 15188 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:33.228147 12646 catalog_manager.cc:5671] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 4 to 6, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 6 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:01:33.204771 15009 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:33.235668 12646 catalog_manager.cc:5671] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 4 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:33.232443 15025 consensus_queue.cc:1048] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:33.221623 15251 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.185878 12883 tablet_service.cc:1558] Processing DeleteTablet for tablet 91ff767ed7c24b97940a459421881055 with delete_type TABLET_DATA_TOMBSTONED (TS 732c3560a688494196d22a1a48f89968 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37306
I20260504 20:01:33.218290 15345 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:33.224757 15327 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:33.248227 12646 catalog_manager.cc:5671] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 6 to 7, leader changed from ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 7 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:33.229158 15106 raft_consensus.cc:2804] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
W20260504 20:01:33.259233 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 93432.5us
I20260504 20:01:33.260718 12646 catalog_manager.cc:5671] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 5 to 6, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 6 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260504 20:01:33.246955 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:33.233829 15338 log.cc:927] Time spent T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260504 20:01:33.223436 15319 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:33.207288 15274 consensus_queue.cc:1059] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.043s
W20260504 20:01:33.268121 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 47254.9us
I20260504 20:01:33.223083 15090 consensus_queue.cc:1048] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20260504 20:01:33.246711 15009 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:33.264709 12646 catalog_manager.cc:5671] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:33.287835 12791 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:33.240939 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:33.291409 12646 catalog_manager.cc:5671] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 4 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:33.248900 15210 consensus_queue.cc:1048] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260504 20:01:33.312402 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:33.299818 12646 catalog_manager.cc:5671] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 3, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:01:33.230633 15246 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.308624 15353 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:33.287143 15319 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:33.323166 12791 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:33.315219 12646 catalog_manager.cc:5671] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 3, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:33.249291 15188 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.329317 12646 catalog_manager.cc:5671] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 4 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:33.289325 15317 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:33.292162 15009 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:33.341917 13497 leader_election.cc:336] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.056s (SENT)
W20260504 20:01:33.343333 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:33.343590 12646 catalog_manager.cc:5671] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:33.302625 15292 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:33.332926 15327 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:33.345588 15349 log.cc:927] Time spent T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260504 20:01:33.277379 15141 consensus_queue.cc:1059] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260504 20:01:33.333981 15319 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:33.361877 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 90470.1us
I20260504 20:01:33.359359 13497 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:33.371573 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:33.372237 12824 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:33.359436 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 73788.4us
I20260504 20:01:33.360132 12646 catalog_manager.cc:5671] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 1 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 4 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:01:33.345631 15350 log.cc:927] Time spent T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260504 20:01:33.374626 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.375653 13499 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:33.341022 15328 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:33.375622 12646 catalog_manager.cc:5671] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 2 to 3, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 3 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:33.346544 15322 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:33.369975 15274 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 20:01:33.387128 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 45293.5us
I20260504 20:01:33.382068 12646 catalog_manager.cc:5671] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 reported cstate change: term changed from 3 to 5, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 5 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:01:33.389268 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 61311.2us
I20260504 20:01:33.390517 12791 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:33.334131 15284 consensus_queue.cc:1059] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260504 20:01:33.324970 15304 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260504 20:01:33.362902 15106 raft_consensus.cc:697] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:33.370422 15327 raft_consensus.cc:697] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:33.386435 15023 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:33.393119 15179 raft_consensus.cc:3055] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:33.412372 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.423816 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.424683 12822 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:33.414760 15251 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:33.417137 12823 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:33.419948 15188 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:33.421507 15295 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:33.394109 15347 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:33.428937 12714 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:33.446823 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 205960us
I20260504 20:01:33.407903 15116 raft_consensus.cc:3055] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:33.411623 15303 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:33.422567 15179 raft_consensus.cc:740] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:33.448726 15343 log.cc:927] Time spent T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.247s user 0.001s sys 0.000s
W20260504 20:01:33.452039 15346 log.cc:927] Time spent T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.248s user 0.001s sys 0.000s
I20260504 20:01:33.453388 15347 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:33.452010 15116 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:33.483667 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.454319 15295 raft_consensus.cc:697] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:33.461400 15339 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:33.477407 14899 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:33.500522 14899 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.506734 15179 consensus_queue.cc:260] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:33.508644 15179 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:33.514138 15246 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:33.516186 15116 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.482585 15347 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:33.518074 15116 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:33.528875 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 58687.6us
I20260504 20:01:33.537902 12865 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:33.518915 15362 raft_consensus.cc:3055] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:33.519711 15339 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:33.550855 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:33.522817 14899 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:33.555694 12816 raft_consensus.cc:1240] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:33.536495 15025 raft_consensus.cc:3055] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
W20260504 20:01:33.560665 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 59473us
I20260504 20:01:33.261838 15345 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.554333 15339 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:33.552366 15362 raft_consensus.cc:740] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:33.558485 15304 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:33.546717 15106 consensus_queue.cc:237] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.566668 15090 consensus_queue.cc:1059] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
W20260504 20:01:33.607445 15359 log.cc:927] Time spent T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260504 20:01:33.520946 15327 consensus_queue.cc:237] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.340327 15353 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:33.614022 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 223660us
I20260504 20:01:33.545035 15274 consensus_queue.cc:1059] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260504 20:01:33.562722 15025 raft_consensus.cc:740] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:33.643772 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 56060.9us
W20260504 20:01:33.609433 15356 log.cc:927] Time spent T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.300s user 0.004s sys 0.000s
W20260504 20:01:33.707722 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:33.748232 15023 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:33.766527 15025 consensus_queue.cc:260] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.548159 15364 consensus_queue.cc:1048] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.213s
I20260504 20:01:33.768097 15025 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:33.768857 15366 log.cc:927] Time spent T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20260504 20:01:33.767457 15023 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.776571 15023 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:33.779080 15345 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:33.785295 15371 log.cc:927] Time spent T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20260504 20:01:33.640879 15287 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:33.748368 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 93843.5us
I20260504 20:01:33.748221 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:33.810613 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 62520us
I20260504 20:01:33.639334 15322 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:33.661459 12823 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:33.764637 15009 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:33.792558 15307 consensus_meta.cc:287] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.667s user 0.004s sys 0.007s
I20260504 20:01:33.648773 15304 raft_consensus.cc:697] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:33.828545 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 347813us
I20260504 20:01:33.821007 12791 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:33.757025 15292 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:33.611363 15295 consensus_queue.cc:237] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.813714 15317 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election won for term 5
W20260504 20:01:33.855548 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:33.914548 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 93681.7us
W20260504 20:01:33.833408 15290 consensus_meta.cc:287] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.714s user 0.004s sys 0.012s
I20260504 20:01:33.830003 15373 tablet_replica.cc:333] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: stopping tablet replica
I20260504 20:01:33.817886 15287 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.837862 15188 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.096s
I20260504 20:01:33.818655 15141 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:33.927032 12824 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.554s user 0.010s sys 0.006s
W20260504 20:01:33.929431 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:33.983758 12791 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:33.912361 15248 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:33.814252 15364 consensus_queue.cc:1048] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.172s
W20260504 20:01:33.991942 15370 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20260504 20:01:33.929589 15328 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:34.044631 12824 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
W20260504 20:01:34.043210 12716 leader_election.cc:336] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.582s (SENT)
I20260504 20:01:34.055987 12819 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260504 20:01:34.047472 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 63911.3us
I20260504 20:01:33.826526 15322 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.827917 15319 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:34.051219 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 297905us
I20260504 20:01:34.055927 12716 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:33.916751 15274 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:34.050757 13527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.028s sys 0.049s
I20260504 20:01:34.011505 15339 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.916522 15292 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:33.828948 15362 consensus_queue.cc:260] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.050100 15248 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:33.941025 15304 consensus_queue.cc:237] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.455060 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:34.465978 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 422821us
W20260504 20:01:34.183099 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 425776us
I20260504 20:01:34.454802 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1835) took 1042 ms. Trace:
I20260504 20:01:34.468137 12821 rpcz_store.cc:276] 0504 20:01:33.412234 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:33.412469 (+ 235us) service_pool.cc:224] Handling call
0504 20:01:33.412822 (+ 353us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:33.417693 (+ 4871us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:33.417731 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:33.418417 (+ 686us) log.cc:844] Serialized 52 byte log entry
0504 20:01:33.486654 (+ 68237us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:33.486701 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:33.486770 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:34.454541 (+967771us) raft_consensus.cc:1685] finished
0504 20:01:34.454574 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:34.454730 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17097,"raft.queue_time_us":9028,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":13076,"thread_start_us":67777,"threads_started":1,"wal-append.queue_time_us":202963}
I20260504 20:01:34.052227 15327 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:34.145576 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:34.471832 15392 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.472076 13531 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:34.472456 15392 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.121948 13528 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260504 20:01:34.102514 13527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s user 0.028s sys 0.049s
W20260504 20:01:34.484517 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:34.168850 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:34.148077 15290 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.379176 15292 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:34.252264 15179 consensus_meta.cc:287] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.743s user 0.015s sys 0.002s
W20260504 20:01:34.486392 15384 log.cc:927] Time spent T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.236s user 0.000s sys 0.000s
W20260504 20:01:34.381346 15280 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.164s user 0.006s sys 0.007s
W20260504 20:01:34.375046 15382 log.cc:927] Time spent T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
W20260504 20:01:34.409341 15375 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.291s user 0.000s sys 0.000s
I20260504 20:01:34.488672 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:34.489976 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:34.557945 12821 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:34.486941 15292 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.251473 15317 raft_consensus.cc:697] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:34.377377 15347 consensus_meta.cc:287] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.828s user 0.008s sys 0.001s
I20260504 20:01:34.266420 15009 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:34.476589 15188 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:01:34.556666 15319 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:34.555424 15280 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.119494 15274 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.363207 15353 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:34.147051 15339 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:34.543562 15179 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.480293 15345 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.057482 15141 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.487293 15284 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.566512 15292 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.480573 15025 consensus_queue.cc:1048] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20260504 20:01:34.569372 15009 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:34.091418 15328 raft_consensus.cc:697] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:34.601708 15296 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.602421 15296 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:34.555562 15344 log.cc:927] Time spent T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 1.295s user 0.005s sys 0.012s
I20260504 20:01:34.510188 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 388) took 1272 ms. Trace:
I20260504 20:01:34.374459 15287 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:34.486685 15290 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.479395 15396 consensus_queue.cc:1048] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.098s
I20260504 20:01:34.560547 15322 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:34.560868 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:34.594275 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 114486us
I20260504 20:01:34.571746 15317 consensus_queue.cc:237] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.627074 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1836) took 1097 ms. Trace:
I20260504 20:01:34.687542 12817 rpcz_store.cc:276] 0504 20:01:33.530073 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:33.531251 (+ 1178us) service_pool.cc:224] Handling call
0504 20:01:33.551905 (+ 20654us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:33.559511 (+ 7606us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:33.559543 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:33.565520 (+ 5977us) log.cc:844] Serialized 26 byte log entry
0504 20:01:33.917869 (+352349us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:33.918231 (+ 362us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:33.918332 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:34.626721 (+708389us) raft_consensus.cc:1685] finished
0504 20:01:34.626755 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:34.626971 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10249,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6684,"thread_start_us":274684,"threads_started":1,"wal-append.queue_time_us":485608}
I20260504 20:01:34.687577 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:34.692312 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:34.693023 12748 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:34.593685 15179 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.638827 12821 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:34.674587 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:34.684322 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:34.684832 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 128197us
I20260504 20:01:34.701649 12747 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
W20260504 20:01:34.628703 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 173743us
I20260504 20:01:34.694379 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:34.695089 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:34.703464 12744 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:34.707839 15400 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.708567 15400 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:34.696719 12714 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:34.711474 15246 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:34.712486 15400 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:34.698864 12716 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.531s (SENT) [suppressed 54 similar messages]
I20260504 20:01:34.574532 15347 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:34.699383 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 186130us
I20260504 20:01:34.622619 15307 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.703466 12741 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:34.712935 15246 raft_consensus.cc:697] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:34.774780 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:34.778581 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1834) took 1367 ms. Trace:
I20260504 20:01:34.779206 12819 rpcz_store.cc:276] 0504 20:01:33.411855 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:33.412023 (+ 168us) service_pool.cc:224] Handling call
0504 20:01:34.055666 (+643643us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:34.056674 (+ 1008us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:34.778274 (+721600us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b540014dcd0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:34.778489 (+ 215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1456273792}
I20260504 20:01:34.781144 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:34.700379 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:34.782193 12742 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:34.716725 13499 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:34.603025 15328 consensus_queue.cc:237] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.692319 15339 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:34.760574 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 73133.9us
I20260504 20:01:34.598536 15284 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:34.595908 15345 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:34.771620 12744 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:34.685742 15392 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:34.803985 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 87393.1us
I20260504 20:01:34.802955 13531 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:34.806453 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 107612us
I20260504 20:01:34.801268 12748 raft_consensus.cc:2468] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:34.807845 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:34.808179 12716 leader_election.cc:336] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.991s (SENT)
I20260504 20:01:34.809059 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1783) took 1863 ms. Trace:
I20260504 20:01:34.809482 12745 rpcz_store.cc:276] 0504 20:01:32.945466 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:32.953908 (+ 8442us) service_pool.cc:224] Handling call
0504 20:01:32.954386 (+ 478us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:32.954637 (+ 251us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:32.954663 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:32.955144 (+ 481us) log.cc:844] Serialized 26 byte log entry
0504 20:01:33.100428 (+145284us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:33.100473 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:33.100541 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:34.476736 (+1376195us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b240014c678. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f0159762e94 00007f0159760f9b 00007f01597efd39 00007f01597ef9c5 00007f01597ef00d 00007f015c5314ae 00007f015c548c2c 00007f015c547f56 00007f015c547b78 00007f015c5515c1
0504 20:01:34.799659 (+322923us) raft_consensus.cc:1685] finished
0504 20:01:34.799691 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:34.808985 (+ 9294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4564352,"thread_start_us":144358,"threads_started":1,"wal-append.queue_time_us":206326}
I20260504 20:01:34.699793 13525 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:34.806567 13499 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:34.699810 15327 raft_consensus.cc:697] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:34.800848 12865 leader_election.cc:400] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5.
I20260504 20:01:34.821224 12893 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:34.803927 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 100471us
I20260504 20:01:34.810640 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:34.813856 13499 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:34.814025 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 99686.8us
I20260504 20:01:34.822937 13532 raft_consensus.cc:2393] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:34.821568 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:34.793553 12821 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:34.852211 12895 raft_consensus.cc:1240] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:34.800928 15386 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.774605 15395 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260504 20:01:34.875260 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 53001.8us
W20260504 20:01:34.876027 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 54452.8us
I20260504 20:01:34.826201 15353 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:34.819911 15392 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.881925 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.890571 12824 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:34.882233 12817 raft_consensus.cc:1240] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
W20260504 20:01:34.882831 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 59033.7us
I20260504 20:01:34.882896 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.883661 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.883683 12790 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:34.891606 12825 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:34.773142 15307 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.881137 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:34.887456 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 73731.2us
I20260504 20:01:34.888403 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.827541 15319 raft_consensus.cc:2804] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:34.874663 15395 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:34.897933 15404 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.898192 15179 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:34.898413 15395 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:34.898561 15404 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:34.898226 15319 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:34.899760 15319 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:34.893244 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:34.894260 12863 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:34.827275 15248 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:34.888414 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.907593 13531 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:34.852617 15381 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.848600 15400 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:34.874812 15280 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:34.850931 15387 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:34.828320 15339 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:34.962716 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.962455 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.963649 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.963945 12821 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:34.927820 13499 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:34.908504 15410 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:34.954442 15389 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260504 20:01:34.954697 15411 raft_consensus.cc:2804] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:34.955137 15210 consensus_queue.cc:1048] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:34.958243 15415 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:34.888906 15403 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:34.884963 15353 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:34.921099 15397 log.cc:927] Time spent T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260504 20:01:34.924930 15296 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:34.950167 15274 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:34.916785 15295 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:01:34.974386 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.011682 12816 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:34.978003 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 50286us
W20260504 20:01:34.985754 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 92254.6us
I20260504 20:01:34.885756 15386 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:35.012770 13499 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:34.923727 15361 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:34.916879 15246 consensus_queue.cc:237] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:34.921176 15388 log.cc:927] Time spent T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.228s user 0.000s sys 0.000s
I20260504 20:01:35.030153 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:35.040661 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:34.903829 15392 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:35.001768 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.980024 15411 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:34.975759 15141 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:35.014568 15290 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.014703 15284 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:35.016620 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:34.966955 15400 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:34.983325 15353 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:35.026232 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:34.965816 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 62329.1us
I20260504 20:01:35.010442 15025 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:35.037894 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:34.921492 15405 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20260504 20:01:35.062853 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:35.083873 12895 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:35.062773 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.017638 15339 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:35.064250 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:35.000769 15292 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.064905 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:35.018010 15352 log.cc:927] Time spent T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 1.701s user 0.015s sys 0.017s
I20260504 20:01:35.069278 12822 raft_consensus.cc:2393] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 2: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:35.069314 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:35.068032 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.026419 15347 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.036146 15361 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:35.074298 12713 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.013s sys 0.025s
I20260504 20:01:34.992377 15396 consensus_queue.cc:1059] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260504 20:01:35.079473 12825 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:34.928140 15381 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:35.029493 15210 consensus_queue.cc:1059] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20260504 20:01:35.042202 15274 consensus_queue.cc:1059] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
I20260504 20:01:34.987952 15403 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:34.978242 15327 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:34.984707 15410 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:34.985612 15415 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:35.084065 12823 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:35.537114 12820 raft_consensus.cc:2410] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:35.537541 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.537871 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:35.538694 12713 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.082s user 0.016s sys 0.055s
I20260504 20:01:35.538928 12819 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:35.539474 12745 raft_consensus.cc:3060] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:35.142717 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 61970.3us
W20260504 20:01:35.110877 12716 leader_election.cc:336] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.073s (SENT)
I20260504 20:01:35.541957 12716 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:35.279422 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 243364us
W20260504 20:01:35.535457 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:35.502048 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:35.063452 15280 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:34.615094 13527 rpcz_store.cc:276] 0504 20:01:33.238088 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:33.247122 (+ 9034us) service_pool.cc:224] Handling call
0504 20:01:33.295648 (+ 48526us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:33.334368 (+ 38720us) raft_consensus.cc:1534] Early marking committed up to index 7
0504 20:01:33.441245 (+106877us) log.cc:844] Serialized 14 byte log entry
0504 20:01:34.485376 (+1044131us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:34.486186 (+ 810us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:34.486460 (+ 274us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:34.486731 (+ 271us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:34.498080 (+ 11349us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":143906,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":1688,"raft.queue_time_us":66782,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":1091584,"threads_started":2}
I20260504 20:01:35.538861 12816 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:35.538292 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.575085 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.065245 15414 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:35.061748 15411 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:35.578423 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 497967us
I20260504 20:01:35.580392 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.581192 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.589320 12865 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:35.589982 12821 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:35.077332 15284 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:35.206311 15303 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:35.630167 15303 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:35.632987 15303 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.350243 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:35.140626 15404 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:35.622445 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 511616us
I20260504 20:01:35.634001 13525 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:35.634068 15404 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:35.634671 15404 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:35.635198 12716 leader_election.cc:336] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.195s (SENT)
I20260504 20:01:35.278147 15025 raft_consensus.cc:697] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:35.635684 15251 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:35.636293 15251 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:35.636816 15404 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.591069 15280 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:35.638688 15280 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:35.624154 15284 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:35.638880 15251 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.579910 15009 raft_consensus.cc:3055] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:35.626212 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1864) took 2529 ms. Trace:
I20260504 20:01:35.136497 15369 raft_consensus.cc:3055] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:35.135458 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:35.636478 15025 consensus_queue.cc:237] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:35.686679 12818 rpcz_store.cc:276] 0504 20:01:33.096738 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:33.096838 (+ 100us) service_pool.cc:224] Handling call
0504 20:01:33.097207 (+ 369us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:33.104697 (+ 7490us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:33.104730 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:33.129765 (+ 25035us) log.cc:844] Serialized 52 byte log entry
0504 20:01:33.257970 (+128205us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:33.258438 (+ 468us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:33.258509 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:34.800274 (+1541765us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:35.625875 (+825601us) raft_consensus.cc:1685] finished
0504 20:01:35.625900 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:35.626137 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5866,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":5353,"spinlock_wait_cycles":25114368,"thread_start_us":89171,"threads_started":1,"wal-append.queue_time_us":151938}
I20260504 20:01:35.641116 12715 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:35.641374 12791 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.006s sys 0.018s
W20260504 20:01:35.705492 12791 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.706s user 0.007s sys 0.018s
W20260504 20:01:35.705941 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 706888us
W20260504 20:01:35.707242 12791 leader_election.cc:336] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.951s (SENT)
I20260504 20:01:35.707826 12791 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:35.126511 15413 raft_consensus.cc:2804] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:35.642799 12864 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.023s sys 0.015s
W20260504 20:01:35.719986 12864 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.958s user 0.023s sys 0.015s
I20260504 20:01:35.607918 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.644155 12825 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:35.549918 15361 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:35.537197 15307 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.549927 15290 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:35.635730 12716 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:35.643242 13497 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.020s sys 0.013s
W20260504 20:01:35.729453 13497 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.703s user 0.020s sys 0.013s
I20260504 20:01:35.616904 15274 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:35.233561 15345 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.755946 15345 consensus_queue.cc:1059] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260504 20:01:35.702253 15358 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.496s user 0.001s sys 0.001s
I20260504 20:01:35.654565 15009 raft_consensus.cc:740] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:35.703666 15363 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260504 20:01:35.719188 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 78165.4us
I20260504 20:01:35.721762 12745 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
W20260504 20:01:35.616638 15409 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.545s user 0.001s sys 0.000s
I20260504 20:01:35.639875 15251 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:35.389009 15398 consensus_queue.cc:1048] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.306s
I20260504 20:01:35.617751 15396 raft_consensus.cc:3055] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
W20260504 20:01:35.729868 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 94706.3us
W20260504 20:01:35.688174 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 98935.7us
W20260504 20:01:35.729837 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 703640us
I20260504 20:01:35.636273 15403 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:35.586077 15339 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:35.737797 12864 connection.cc:254] RPC call timeout handler was delayed by -1.617836792999924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
I20260504 20:01:35.743001 15284 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:35.695005 15210 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:35.719424 13529 raft_consensus.cc:1813] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Handling vote request from an unknown peer 732c3560a688494196d22a1a48f89968
I20260504 20:01:35.695982 15369 raft_consensus.cc:740] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:35.745982 15423 consensus_queue.cc:1059] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:01:35.747287 12895 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:01:35.748202 15417 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:01:35.749857 15429 log.cc:927] Time spent T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260504 20:01:35.754181 15372 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:35.774935 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 67726.2us
I20260504 20:01:35.716955 12823 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:35.773356 15405 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:35.772588 15386 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:35.791046 13497 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:35.777570 15392 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.776001 15400 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:35.761840 15290 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:36.069885 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:36.224448 12749 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260504 20:01:36.316533 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 425) took 1279 ms. Trace:
W20260504 20:01:36.269423 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 688015us
W20260504 20:01:36.269721 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:36.030100 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 446518us
I20260504 20:01:36.318133 12823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 1848) took 1273 ms. Trace:
I20260504 20:01:36.068365 12742 raft_consensus.cc:1275] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:36.318608 12823 rpcz_store.cc:276] 0504 20:01:35.044630 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:35.062567 (+ 17937us) service_pool.cc:224] Handling call
0504 20:01:36.318065 (+1255498us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:36.275171 12746 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260504 20:01:35.842391 12863 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.005s sys 0.026s
W20260504 20:01:36.308300 12819 consensus_meta.cc:287] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.758s user 0.014s sys 0.000s
W20260504 20:01:36.230058 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 443823us
I20260504 20:01:36.067451 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:36.321453 12716 leader_election.cc:336] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.868s (SENT)
I20260504 20:01:36.321821 12744 raft_consensus.cc:2393] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:36.322019 12716 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:36.069118 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:36.320600 12819 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
W20260504 20:01:36.323360 12819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1879) took 1354 ms (client timeout 1584 ms). Trace:
I20260504 20:01:36.323933 12744 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260504 20:01:36.324383 12744 pending_rounds.cc:85] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Aborting all ops after (but not including) 3
I20260504 20:01:36.324677 12744 pending_rounds.cc:107] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260504 20:01:36.325461 12744 raft_consensus.cc:2889] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260504 20:01:36.016463 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:36.324595 12716 leader_election.cc:336] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.225s (SENT)
W20260504 20:01:36.328418 12716 leader_election.cc:336] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.825s (SENT)
W20260504 20:01:36.276772 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 480859us
I20260504 20:01:36.068825 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:36.315500 12791 leader_election.cc:336] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.997s (SENT)
I20260504 20:01:36.326673 12750 raft_consensus.cc:2393] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:36.317098 12895 rpcz_store.cc:276] 0504 20:01:35.037084 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:35.037540 (+ 456us) service_pool.cc:224] Handling call
0504 20:01:36.316354 (+1278814us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:36.331086 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:36.320094 12863 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.098s user 0.016s sys 0.039s
W20260504 20:01:36.323730 12819 rpcz_store.cc:269] 0504 20:01:34.985284 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:35.535141 (+549857us) service_pool.cc:224] Handling call
0504 20:01:36.323297 (+788156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:36.335388 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:36.336249 12895 raft_consensus.cc:2368] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7.
I20260504 20:01:36.337034 12820 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:36.330933 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:36.338244 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.338738 12893 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:36.339520 12895 raft_consensus.cc:2410] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:01:36.345148 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.345351 12714 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:36.346709 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:36.348073 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:35.782127 15307 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:36.348825 12892 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:36.284696 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1851) took 1212 ms. Trace:
I20260504 20:01:36.351878 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:36.353363 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:36.355266 12748 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:36.354316 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.354914 12822 rpcz_store.cc:276] 0504 20:01:35.072282 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:35.589527 (+517245us) service_pool.cc:224] Handling call
0504 20:01:35.616693 (+ 27166us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:35.616934 (+ 241us) raft_consensus.cc:1534] Early marking committed up to index 7
0504 20:01:35.788732 (+171798us) log.cc:844] Serialized 14 byte log entry
0504 20:01:36.245740 (+457008us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:36.284025 (+ 38285us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:36.284283 (+ 258us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:36.284400 (+ 117us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:36.284605 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19802,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":102,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"thread_start_us":621348,"threads_started":2}
I20260504 20:01:36.359404 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.359404 12714 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:36.359421 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:36.360131 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 569147us
W20260504 20:01:36.239601 15319 consensus_meta.cc:287] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.330s user 0.014s sys 0.000s
I20260504 20:01:35.785913 15396 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:35.872896 15409 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260504 20:01:35.783715 15274 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
W20260504 20:01:36.386183 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:36.494746 15432 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:36.520845 13497 connection.cc:254] RPC call timeout handler was delayed by -1.447603756000035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2525s for a response
W20260504 20:01:36.480409 12716 leader_election.cc:336] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.503s (SENT)
I20260504 20:01:36.491950 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:36.491981 13529 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:36.522306 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:35.787161 15251 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:36.351972 15364 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:36.117362 15420 log.cc:927] Time spent T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.587s user 0.001s sys 0.000s
W20260504 20:01:35.792505 15304 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.085s sys 0.112s
W20260504 20:01:36.073052 15407 log.cc:927] Time spent T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 1.364s user 0.003s sys 0.006s
I20260504 20:01:36.384485 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.855559 15347 raft_consensus.cc:3055] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:36.435132 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:36.448966 12747 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:01:36.455857 12741 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:36.360630 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.448074 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.389851 12864 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:36.630918 15353 consensus_meta.cc:287] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.491s user 0.013s sys 0.002s
I20260504 20:01:36.387017 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:36.525643 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:36.361907 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:36.036541 15395 consensus_meta.cc:287] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.138s user 0.024s sys 0.016s
I20260504 20:01:36.434829 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:36.337751 15116 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:36.641245 15304 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.552s user 0.099s sys 0.142s
I20260504 20:01:36.535089 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:35.857661 15381 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:35.791673 15415 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:36.387554 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:36.048928 15413 raft_consensus.cc:697] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:36.522536 15340 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:36.447808 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.623199 12716 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:36.624500 13497 connection.cc:254] RPC call timeout handler was delayed by -1.0031955440000502s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
I20260504 20:01:36.220937 15322 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:36.350852 15417 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:36.337538 15287 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:36.103641 15295 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:36.337471 15090 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:36.629715 12898 raft_consensus.cc:2410] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:36.535466 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:35.856506 15410 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:36.535676 15369 consensus_queue.cc:260] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:36.634938 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:36.635952 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 263131us
W20260504 20:01:36.638872 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 323395us
I20260504 20:01:36.219919 15414 raft_consensus.cc:697] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:36.643868 12823 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:36.464617 15405 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:36.069499 13527 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60"
dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:55682
I20260504 20:01:35.795089 15363 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.832s
I20260504 20:01:36.381019 15025 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:36.371966 15392 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:36.530835 15423 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 20:01:36.650511 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 260794us
I20260504 20:01:36.534359 15332 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:36.371120 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:36.658133 12743 raft_consensus.cc:1275] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260504 20:01:36.527032 12893 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:36.661214 12825 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:36.786661 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.640527 13529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40940 (request call id 320) took 2585 ms. Trace:
W20260504 20:01:36.899449 13497 leader_election.cc:336] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.556s (SENT)
W20260504 20:01:36.900084 12791 leader_election.cc:336] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.045s (SENT)
I20260504 20:01:37.003127 12791 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:36.900851 13499 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:36.357031 15210 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:36.634087 13525 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:36.843685 12863 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:35.862169 15424 consensus_queue.cc:1059] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.789s
W20260504 20:01:36.871408 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:36.727481 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:35.866961 15009 consensus_queue.cc:260] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:36.357367 15372 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:36.648206 15347 raft_consensus.cc:740] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:37.003088 13497 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:36.846575 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1888) took 1775 ms. Trace:
I20260504 20:01:37.003118 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:36.920744 12818 raft_consensus.cc:1240] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
W20260504 20:01:37.004499 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 524126us
I20260504 20:01:36.731639 15380 raft_consensus.cc:3055] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:37.049883 12891 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:36.784341 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:36.919701 13527 raft_consensus.cc:606] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:36.794914 15345 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:36.904007 15417 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:36.904197 15295 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
W20260504 20:01:37.022087 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 178514us
W20260504 20:01:37.022336 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 121586us
I20260504 20:01:36.731294 15274 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:36.733633 15290 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:36.631675 15319 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.043308 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:37.047972 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 148562us
I20260504 20:01:36.638231 15364 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:36.681151 15360 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:37.096050 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:55682 (request call id 0) took 1018 ms. Trace:
I20260504 20:01:37.096637 15360 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:36.650271 15246 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:37.097254 15360 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:36.697883 15353 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.031844 15332 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.100528 15353 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:36.914875 15116 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.102727 15434 raft_consensus.cc:993] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: : Instructing follower 82787d8e64d24ca885ca517aa8ed1be2 to start an election
I20260504 20:01:37.102999 15116 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:37.103410 15434 raft_consensus.cc:1081] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Signalling peer 82787d8e64d24ca885ca517aa8ed1be2 to start an election
I20260504 20:01:37.068830 15317 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:36.917371 15395 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.105000 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:37.096748 13527 rpcz_store.cc:276] 0504 20:01:36.056185 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.069326 (+ 13141us) service_pool.cc:224] Handling call
0504 20:01:37.074132 (+1004806us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:37.107159 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:36.642444 15432 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:37.051728 13496 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.011s sys 0.080s
I20260504 20:01:37.111471 15319 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:36.913081 15090 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.112510 15414 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.114424 15090 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:36.725378 15251 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:36.921015 15392 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:37.116289 13531 raft_consensus.cc:1240] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:37.052309 15399 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:37.026996 15363 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:37.061455 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 161406us
W20260504 20:01:37.123540 12791 leader_election.cc:336] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.170s (SENT)
I20260504 20:01:37.053166 15381 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:37.010576 15340 raft_consensus.cc:697] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:37.076053 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:37.126461 12891 raft_consensus.cc:2368] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:01:37.056749 13524 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
W20260504 20:01:37.115821 15411 consensus_meta.cc:287] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.417s user 0.014s sys 0.000s
I20260504 20:01:37.080060 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:37.082262 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:37.012128 15304 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260504 20:01:37.091033 15345 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.121563 15392 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:37.110519 12892 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:01:37.058182 12716 leader_election.cc:336] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.531s (SENT)
I20260504 20:01:37.147603 12716 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:37.148963 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 90810.7us
I20260504 20:01:37.007117 12896 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:37.047801 15287 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:36.932801 15369 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:37.049592 12820 rpcz_store.cc:276] 0504 20:01:35.070897 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:35.583170 (+512273us) service_pool.cc:224] Handling call
0504 20:01:36.317845 (+734675us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:36.361827 (+ 43982us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:36.846248 (+484421us) spinlock_profiling.cc:243] Waited 730 ms on lock 0x7b5400300cd0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:36.846475 (+ 227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1678432640}
I20260504 20:01:37.161528 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:36.914188 15322 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.052093 15423 raft_consensus.cc:2804] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:37.052860 15380 raft_consensus.cc:740] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:37.125411 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:37.079239 15425 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.438s user 0.000s sys 0.001s
I20260504 20:01:37.131122 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.177073 12747 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:01:37.095930 15364 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:37.002679 13529 rpcz_store.cc:276] 0504 20:01:34.048553 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:34.049097 (+ 544us) service_pool.cc:224] Handling call
0504 20:01:35.135229 (+1086132us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:36.634192 (+1498963us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59562368}
I20260504 20:01:37.142753 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:37.112138 15432 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:37.101605 15332 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:37.146178 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:36.935312 15025 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.152155 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:37.153586 12716 leader_election.cc:336] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.181s (SENT)
W20260504 20:01:37.157161 13528 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.021s sys 0.020s
I20260504 20:01:37.179287 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:37.200449 13528 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.022s sys 0.021s
I20260504 20:01:37.097635 15246 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.180588 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:37.133760 15399 raft_consensus.cc:697] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:37.206264 12747 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:37.206635 12896 raft_consensus.cc:2393] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:37.201505 12716 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:37.164604 15251 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:37.208434 12864 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:37.182777 15362 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:37.156260 13526 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:37.198417 15395 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:37.220260 15445 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260504 20:01:37.181133 12741 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:57934
I20260504 20:01:37.222293 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:37.226325 12747 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:37.246968 12741 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:01:37.143733 15411 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.230731 12893 raft_consensus.cc:2410] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:01:37.125624 15413 consensus_queue.cc:237] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.239362 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:37.250146 13499 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:37.198170 15423 raft_consensus.cc:697] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
W20260504 20:01:37.243786 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 90242.1us
I20260504 20:01:37.250531 12896 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:37.247419 12741 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:37.179430 15363 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.254766 15380 consensus_queue.cc:260] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.256666 15363 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:37.259747 12716 leader_election.cc:336] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 3.711s (SENT)
I20260504 20:01:37.260416 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:37.264516 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 353555us
I20260504 20:01:37.244814 15432 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:37.279440 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 141084us
I20260504 20:01:37.280094 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.270845 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.273379 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:37.292994 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 223539us
I20260504 20:01:37.283378 12816 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:37.254156 15307 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:37.286701 12716 leader_election.cc:336] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.692s (SENT)
I20260504 20:01:37.294482 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:37.297292 12716 leader_election.cc:336] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.737s (SENT)
I20260504 20:01:37.297964 12716 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:37.278838 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 201143us
I20260504 20:01:37.282989 12897 raft_consensus.cc:1240] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
W20260504 20:01:37.299391 13497 leader_election.cc:336] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.041s (SENT)
I20260504 20:01:37.300014 13497 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:37.301744 13497 leader_election.cc:336] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.153s (SENT)
I20260504 20:01:37.302109 12863 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:37.304991 12790 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:37.282891 12741 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.284117 13499 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:37.271981 15319 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260504 20:01:37.329998 12716 leader_election.cc:336] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.835s (SENT)
I20260504 20:01:37.330662 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:37.332103 12716 leader_election.cc:336] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.897s (SENT)
I20260504 20:01:37.332583 12716 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:37.308810 15210 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:37.229238 15340 consensus_queue.cc:237] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.334370 13528 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260504 20:01:37.309239 15454 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:37.337319 15439 raft_consensus.cc:2804] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:37.337984 15439 raft_consensus.cc:697] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:37.324265 12741 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:37.325340 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:37.345535 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:37.352726 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.337051 15295 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.354117 12863 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:37.357025 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:37.358119 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:37.343694 15450 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.220s user 0.001s sys 0.000s
I20260504 20:01:37.344136 15319 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:37.358771 12825 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260504 20:01:37.357666 15116 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:37.350082 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:37.360198 15339 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:37.367234 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1842) took 1024 ms. Trace:
I20260504 20:01:37.373446 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:37.374329 12716 leader_election.cc:336] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.905s (SENT)
I20260504 20:01:37.322067 15424 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260504 20:01:37.382719 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:37.343346 15451 log.cc:927] Time spent T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
W20260504 20:01:37.342842 15443 log.cc:927] Time spent T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.219s user 0.001s sys 0.000s
I20260504 20:01:37.384900 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:37.350289 15251 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260504 20:01:37.437585 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 49358.3us
W20260504 20:01:37.438910 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 51199.3us
I20260504 20:01:37.439244 12898 rpcz_store.cc:276] 0504 20:01:36.342760 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.342896 (+ 136us) service_pool.cc:224] Handling call
0504 20:01:37.366969 (+1024073us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54000f3a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:37.367170 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":630492928}
I20260504 20:01:37.440279 12716 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:37.343384 15317 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:37.351145 15307 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:37.351462 15453 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:37.442960 12822 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:37.383854 15361 consensus_meta.cc:287] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.617s user 0.015s sys 0.001s
I20260504 20:01:37.346710 15372 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:37.445443 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 140587us
I20260504 20:01:37.405015 15364 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:01:37.436952 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:37.437637 15449 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:37.454596 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 80318.7us
I20260504 20:01:37.455660 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:37.455899 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.327698 15461 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:37.391996 15319 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:37.523651 12716 leader_election.cc:336] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.917s (SENT)
I20260504 20:01:37.524122 12716 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:37.522950 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:37.454578 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:37.456521 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 172539us
I20260504 20:01:37.346930 15363 consensus_queue.cc:1059] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
I20260504 20:01:37.468631 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:37.451138 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 128426us
I20260504 20:01:37.472182 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:37.335453 15396 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:37.489235 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:37.529392 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:37.462332 15390 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:37.532653 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:37.455541 15317 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:37.559840 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:37.570003 12821 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:37.570887 12825 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:37.561183 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.571743 12714 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:37.572383 12741 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:37.565276 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.329280 15399 consensus_queue.cc:237] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.461045 15411 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:37.568288 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:37.568957 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.582511 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.471603 15364 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:37.590114 12747 raft_consensus.cc:1240] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:37.573683 12898 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:37.445894 15290 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.084s
I20260504 20:01:37.430670 15460 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
W20260504 20:01:37.577067 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 53467.9us
W20260504 20:01:37.616254 12716 leader_election.cc:336] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.924s (SENT)
I20260504 20:01:37.616695 12716 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:37.616801 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:37.592867 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.617425 12747 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:37.617861 12891 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:37.560382 13530 raft_consensus.cc:1240] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:37.618804 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.595605 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.620095 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.620915 12745 raft_consensus.cc:2410] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260504 20:01:37.576202 15317 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:37.622691 12748 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:37.624284 15347 consensus_queue.cc:260] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.594278 15141 raft_consensus.cc:3055] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:37.625677 15141 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:37.626636 15141 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:37.627898 15141 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:37.629297 12820 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260504 20:01:37.630002 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:37.621225 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:37.638075 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.609143 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:37.615460 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 160961us
I20260504 20:01:37.648283 12898 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:37.553318 15461 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:37.638888 12746 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
W20260504 20:01:37.650730 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 79128.1us
I20260504 20:01:37.651202 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.652050 12746 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:37.653223 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.655885 12746 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:37.658105 12746 raft_consensus.cc:1240] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:37.655803 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.659409 12898 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:37.667302 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.668895 12741 raft_consensus.cc:1813] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Handling vote request from an unknown peer 732c3560a688494196d22a1a48f89968
I20260504 20:01:37.669560 12741 raft_consensus.cc:2410] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260504 20:01:37.671720 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.575309 15025 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:37.674594 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:37.587313 15307 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:37.678754 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:37.679091 12750 raft_consensus.cc:1240] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:37.599927 15364 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:37.681318 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.681712 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:37.683444 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 67230us
I20260504 20:01:37.683485 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:37.684628 12716 leader_election.cc:336] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.099s (SENT)
I20260504 20:01:37.685062 12716 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:37.684446 12820 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:37.339035 15439 consensus_queue.cc:237] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.682096 12898 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:37.682405 12891 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:01:37.697218 13499 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:37.697608 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.698652 12891 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:37.468192 15361 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.704437 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.669608 15322 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:37.711059 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:37.682762 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:37.710623 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.714020 12750 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:37.674203 15116 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:37.666601 15461 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:37.447690 15423 consensus_queue.cc:237] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.729388 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:37.667763 15210 consensus_queue.cc:1059] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260504 20:01:37.655812 15467 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:37.739188 12823 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260504 20:01:37.567543 15449 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.705258 15332 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:37.744655 12820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:37.735073 15322 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:37.748822 15473 log.cc:927] Time spent T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260504 20:01:37.724897 12747 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:37.766963 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:37.750754 12898 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:37.469193 15453 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.725461 15116 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:37.723488 15390 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:37.783509 12820 raft_consensus.cc:606] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:38.011783 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 362797us
I20260504 20:01:38.860827 12820 raft_consensus.cc:614] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:37.779013 15469 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:37.883353 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1793) took 1815 ms. Trace:
W20260504 20:01:37.862042 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:38.001379 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 252139us
I20260504 20:01:38.274094 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:38.863428 12741 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
W20260504 20:01:38.874547 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:38.005075 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 332407us
I20260504 20:01:38.864346 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:38.875314 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:38.235082 15210 consensus_queue.cc:1243] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer ef6057d9a49e4942b47ea0fcd66ab6f7 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260504 20:01:37.891542 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:38.052987 15361 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:38.235204 12744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.013s sys 0.045s
W20260504 20:01:38.888504 12744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.157s user 0.025s sys 0.054s
W20260504 20:01:38.340235 15317 consensus_meta.cc:287] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.718s user 0.008s sys 0.004s
W20260504 20:01:37.935882 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:38.866066 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:38.048584 15392 consensus_meta.cc:287] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.875s user 0.005s sys 0.008s
I20260504 20:01:38.929895 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:37.777930 15295 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:38.237017 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:37.854377 15449 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:38.379626 13533 raft_consensus.cc:3055] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
W20260504 20:01:38.592517 12816 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.925s user 0.005s sys 0.009s
W20260504 20:01:38.040609 15447 log.cc:927] Time spent T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.917s user 0.002s sys 0.000s
W20260504 20:01:38.063304 15360 consensus_meta.cc:287] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.965s user 0.014s sys 0.000s
I20260504 20:01:38.991326 13533 raft_consensus.cc:740] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:37.878588 15246 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:38.864398 12898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1868) took 1704 ms (client timeout 2191 ms). Trace:
W20260504 20:01:39.012379 12898 rpcz_store.cc:269] 0504 20:01:37.160419 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.690707 (+530288us) service_pool.cc:224] Handling call
0504 20:01:38.864335 (+1173628us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:38.301713 13531 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 398) took 1930 ms. Trace:
I20260504 20:01:39.013710 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:39.014803 12898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1881) took 1655 ms (client timeout 2099 ms). Trace:
I20260504 20:01:39.013844 13531 rpcz_store.cc:276] 0504 20:01:36.371259 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.371643 (+ 384us) service_pool.cc:224] Handling call
0504 20:01:36.494834 (+123191us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:37.116828 (+621994us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:38.301321 (+1184493us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b54004902d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:38.301610 (+ 289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1428569472}
W20260504 20:01:39.015405 12898 rpcz_store.cc:269] 0504 20:01:37.359456 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.013388 (+1653932us) service_pool.cc:224] Handling call
0504 20:01:39.014731 (+ 1343us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.016521 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:38.874476 12894 raft_consensus.cc:2393] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:01:38.883394 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:38.188535 15332 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:38.062709 15369 consensus_meta.cc:287] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.884s user 0.011s sys 0.003s
W20260504 20:01:37.855937 15474 log.cc:927] Time spent T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
W20260504 20:01:39.026193 12894 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1872) took 1768 ms (client timeout 2181 ms). Trace:
W20260504 20:01:39.026959 12894 rpcz_store.cc:269] 0504 20:01:37.257640 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:38.273666 (+1016026us) service_pool.cc:224] Handling call
0504 20:01:39.026067 (+752401us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.028546 12890 raft_consensus.cc:2410] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:39.028115 13533 consensus_queue.cc:260] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:37.776242 15363 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:38.332224 12893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1855) took 1983 ms (client timeout 2223 ms). Trace:
I20260504 20:01:39.029342 15363 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:39.029836 12893 rpcz_store.cc:269] 0504 20:01:36.348712 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.077645 (+728933us) service_pool.cc:224] Handling call
0504 20:01:38.331886 (+1254241us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54003ba454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:38.332119 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":286501760}
I20260504 20:01:39.031520 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.032612 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1901) took 1508 ms. Trace:
I20260504 20:01:39.011179 15295 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260504 20:01:39.033099 12893 rpcz_store.cc:276] 0504 20:01:37.524097 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.031107 (+1507010us) service_pool.cc:224] Handling call
0504 20:01:39.032534 (+ 1427us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.033763 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:39.031522 15363 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:39.034508 13531 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Advancing to term 9
I20260504 20:01:39.035564 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:39.036744 12893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1878) took 1678 ms (client timeout 2224 ms). Trace:
W20260504 20:01:39.037300 12893 rpcz_store.cc:269] 0504 20:01:37.358245 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.035050 (+1676805us) service_pool.cc:224] Handling call
0504 20:01:39.036659 (+ 1609us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.014600 15360 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.038609 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:39.039688 12893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1849) took 2693 ms (client timeout 3286 ms). Trace:
W20260504 20:01:39.040153 12893 rpcz_store.cc:269] 0504 20:01:36.346878 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.038268 (+2691390us) service_pool.cc:224] Handling call
0504 20:01:39.039612 (+ 1344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.040498 15360 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:38.090823 15417 consensus_meta.cc:287] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.015s user 0.015s sys 0.000s
I20260504 20:01:38.045040 15396 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:38.188469 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 424) took 3196 ms. Trace:
I20260504 20:01:38.863970 15009 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:39.046057 12897 rpcz_store.cc:276] 0504 20:01:34.992516 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:34.994569 (+ 2053us) service_pool.cc:224] Handling call
0504 20:01:35.016183 (+ 21614us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:37.299352 (+2283169us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:38.188025 (+888673us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b54003b9f54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:38.188348 (+ 323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5212801280}
I20260504 20:01:38.872931 15469 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:38.535249 15141 consensus_meta.cc:287] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.907s user 0.008s sys 0.008s
I20260504 20:01:38.856084 15347 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:38.137678 15435 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:38.870267 15322 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:38.898243 15340 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.187s user 0.016s sys 0.021s
W20260504 20:01:37.852988 15463 log.cc:927] Time spent T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.284s user 0.000s sys 0.000s
I20260504 20:01:38.274324 15345 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:37.881704 15411 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:38.042507 15401 raft_consensus.cc:3055] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:38.294260 15477 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:38.872220 15116 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:38.933032 15439 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.004s sys 0.010s
I20260504 20:01:37.782449 15454 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 1.182s
I20260504 20:01:39.103545 15322 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:39.090947 15340 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.379s user 0.020s sys 0.032s
W20260504 20:01:38.977458 15362 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.748s user 0.013s sys 0.003s
I20260504 20:01:38.963491 15210 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:38.929888 12742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.164s user 0.000s sys 0.037s
W20260504 20:01:38.932658 12746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.183s user 0.011s sys 0.010s
I20260504 20:01:38.873202 12743 rpcz_store.cc:276] 0504 20:01:36.068356 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.069082 (+ 726us) service_pool.cc:224] Handling call
0504 20:01:36.069394 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:37.013178 (+943784us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:37.883074 (+869896us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b540022f6d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:37.883278 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1339532160}
I20260504 20:01:38.864959 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1652) took 1268 ms. Trace:
I20260504 20:01:38.961208 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:38.875123 12741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1666) took 1149 ms. Trace:
W20260504 20:01:38.991922 12816 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.337s user 0.005s sys 0.017s
W20260504 20:01:38.932705 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 47200.2us
I20260504 20:01:39.269773 13527 raft_consensus.cc:1275] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:39.279572 15415 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.280121 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:39.191578 12817 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260504 20:01:39.271118 15364 consensus_meta.cc:287] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.570s user 0.011s sys 0.004s
I20260504 20:01:39.283118 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:39.283879 15364 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:39.285248 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:39.286065 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.287462 12894 raft_consensus.cc:2468] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:01:39.288415 12894 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1890) took 1892 ms (client timeout 2125 ms). Trace:
I20260504 20:01:39.192809 12819 raft_consensus.cc:1275] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260504 20:01:38.930740 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:38.880522 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49576 (request call id 33) took 1127 ms. Trace:
I20260504 20:01:39.278365 15396 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260504 20:01:39.297495 12820 rpcz_store.cc:276] 0504 20:01:37.734509 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.734713 (+ 204us) service_pool.cc:224] Handling call
0504 20:01:38.861436 (+1126723us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":759680}
I20260504 20:01:39.297756 12821 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:38.982298 12823 raft_consensus.cc:1275] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260504 20:01:39.272056 12742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.506s user 0.000s sys 0.038s
I20260504 20:01:38.931502 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:39.199914 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:39.274363 12747 rpcz_store.cc:276] 0504 20:01:37.596922 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.616496 (+ 19574us) service_pool.cc:224] Handling call
0504 20:01:38.864890 (+1248394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.304692 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:39.274755 13533 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:39.306178 12824 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:39.306974 15340 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.307929 15340 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:39.205175 12716 connection.cc:254] RPC call timeout handler was delayed by -1.0589316499999768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
W20260504 20:01:38.054510 12892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1857) took 1705 ms (client timeout 1905 ms). Trace:
W20260504 20:01:38.870461 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:39.310297 12892 rpcz_store.cc:269] 0504 20:01:36.349362 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.359891 (+ 10529us) service_pool.cc:224] Handling call
0504 20:01:38.054204 (+1694313us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b54003b9a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:38.054397 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1060678912}
I20260504 20:01:39.311940 12824 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:39.312659 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:39.271821 15210 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.313047 12824 raft_consensus.cc:606] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
W20260504 20:01:38.933003 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:39.210427 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:39.314841 12893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1902) took 1777 ms (client timeout 2105 ms). Trace:
I20260504 20:01:39.047610 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.315208 15364 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:39.315609 15210 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:38.296624 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:39.315230 12893 rpcz_store.cc:269] 0504 20:01:37.537876 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.041053 (+1503177us) service_pool.cc:224] Handling call
0504 20:01:39.314606 (+273553us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:39.025460 12865 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37318 dropped due to backpressure: service queue is full with 50 items
I20260504 20:01:39.213068 12896 raft_consensus.cc:2410] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260504 20:01:39.017428 12898 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
W20260504 20:01:39.276245 12748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.001s sys 0.001s
W20260504 20:01:39.319208 12898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1894) took 1871 ms (client timeout 2015 ms). Trace:
W20260504 20:01:39.319815 12748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.972s user 0.002s sys 0.001s
W20260504 20:01:39.319867 12898 rpcz_store.cc:269] 0504 20:01:37.448212 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.016212 (+1568000us) service_pool.cc:224] Handling call
0504 20:01:39.318925 (+302713us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.214491 12899 raft_consensus.cc:2410] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260504 20:01:39.057881 13531 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 9.
I20260504 20:01:38.978713 15317 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.312565 15449 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.340636 15449 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.341998 15317 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:38.991950 15319 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.423s user 0.015s sys 0.000s
I20260504 20:01:39.343142 15454 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:39.343601 15319 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.344170 15454 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.125490 15386 raft_consensus.cc:3055] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:39.346518 15460 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:39.346491 15481 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:39.347196 13528 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:39.347114 15400 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:39.091758 15401 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:39.348500 15400 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.318125 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 330) took 2235 ms. Trace:
I20260504 20:01:39.349830 15401 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.096838 15411 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.352136 15400 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:39.276687 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 243838us
I20260504 20:01:39.012162 15405 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.017s
I20260504 20:01:39.354338 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1899) took 2006 ms. Trace:
I20260504 20:01:39.076726 15417 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:38.977870 15392 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.356606 15322 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:39.357474 15322 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.358177 15417 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:39.358371 15392 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.080371 15380 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:39.009958 15432 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.717s user 0.016s sys 0.000s
I20260504 20:01:39.360087 15432 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:39.360455 15322 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:39.361837 12749 raft_consensus.cc:2393] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:39.363209 12745 raft_consensus.cc:1240] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:39.364621 15373 raft_consensus.cc:2243] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Raft consensus shutting down.
W20260504 20:01:39.095495 15439 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s user 0.004s sys 0.011s
I20260504 20:01:38.976161 15441 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:39.367432 15441 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.083812 15453 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:39.129477 15461 consensus_meta.cc:287] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.374s user 0.007s sys 0.003s
I20260504 20:01:39.371438 15461 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.318192 12865 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37318 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1892)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1891)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1698)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1862)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1896)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1717)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 435)
W20260504 20:01:37.852485 15464 log.cc:927] Time spent T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260504 20:01:39.074488 15469 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:39.051867 15332 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.265966 15246 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.095847 15477 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.267390 15390 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:39.269120 13498 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:39.269629 15483 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.260407 15434 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:39.089601 15435 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.262058 15467 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:39.280398 15415 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.278580 12891 raft_consensus.cc:2410] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260504 20:01:39.272492 12746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.523s user 0.012s sys 0.010s
I20260504 20:01:39.275841 12741 rpcz_store.cc:276] 0504 20:01:37.726216 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.726675 (+ 459us) service_pool.cc:224] Handling call
0504 20:01:38.875047 (+1148372us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.275343 12750 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:39.275085 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:39.296456 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1655) took 1678 ms. Trace:
I20260504 20:01:39.270320 15488 consensus_queue.cc:1059] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
W20260504 20:01:39.312450 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:39.316227 12897 raft_consensus.cc:2393] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:01:38.994253 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:39.317505 13525 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40940 (request call id 324) took 2251 ms (client timeout 1922 ms). Trace:
I20260504 20:01:38.988620 13526 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:39.318120 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.265183 12863 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:39.322283 12892 raft_consensus.cc:2410] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260504 20:01:39.271915 12890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1845) took 2926 ms (client timeout 2825 ms). Trace:
I20260504 20:01:39.305658 12747 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
W20260504 20:01:39.289038 12894 rpcz_store.cc:269] 0504 20:01:37.396027 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.027745 (+1631718us) service_pool.cc:224] Handling call
0504 20:01:39.288277 (+260532us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.334576 15179 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.334790 15295 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:39.338013 15025 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.342204 15327 consensus_queue.cc:1048] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:39.349963 13527 rpcz_store.cc:276] 0504 20:01:37.082318 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.108688 (+ 26370us) service_pool.cc:224] Handling call
0504 20:01:37.109530 (+ 842us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:39.279210 (+2169680us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.317639 (+ 38429us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b5400490cd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:39.318023 (+ 384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4966095232}
I20260504 20:01:39.355273 12824 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:39.358410 15340 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.370299 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1657) took 1734 ms. Trace:
I20260504 20:01:39.354811 12816 rpcz_store.cc:276] 0504 20:01:37.347384 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.347610 (+ 226us) service_pool.cc:224] Handling call
0504 20:01:37.348031 (+ 421us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:37.348392 (+ 361us) raft_consensus.cc:1534] Early marking committed up to index 3
0504 20:01:37.560678 (+212286us) log.cc:844] Serialized 14 byte log entry
0504 20:01:39.334524 (+1773846us) log.cc:844] Serialized 14 byte log entry
0504 20:01:39.353562 (+ 19038us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:39.353641 (+ 79us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:39.353739 (+ 98us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.353875 (+ 136us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:39.354100 (+ 225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65396,"prepare.queue_time_us":341156,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":207967,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":22831,"thread_start_us":1923947,"threads_started":4,"wal-append.queue_time_us":126535}
W20260504 20:01:39.378465 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 98729.3us
W20260504 20:01:39.424304 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 52437.9us
I20260504 20:01:39.221535 12646 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39316 (request call id 226) took 9561 ms. Trace:
I20260504 20:01:39.445531 13533 raft_consensus.cc:2468] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:01:39.449198 12743 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:39.376467 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:39.408959 12899 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1860) took 2776 ms (client timeout 1531 ms). Trace:
W20260504 20:01:39.451370 12899 rpcz_store.cc:269] 0504 20:01:36.632553 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.650778 (+ 18225us) service_pool.cc:224] Handling call
0504 20:01:39.408557 (+2757779us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b54002f71d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:39.408836 (+ 279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5029618944}
W20260504 20:01:39.409574 12896 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1867) took 2275 ms (client timeout 2105 ms). Trace:
W20260504 20:01:39.441592 12890 rpcz_store.cc:269] 0504 20:01:36.345062 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.345203 (+ 141us) service_pool.cc:224] Handling call
0504 20:01:39.271107 (+2925904us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b540013c9d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:39.271511 (+ 404us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4891593984}
I20260504 20:01:39.416998 13530 raft_consensus.cc:1240] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:39.378122 12716 leader_election.cc:336] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.574s (SENT)
W20260504 20:01:39.458097 12893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1903) took 1894 ms (client timeout 2131 ms). Trace:
W20260504 20:01:39.454173 12747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1675) took 1408 ms (client timeout 1735 ms). Trace:
W20260504 20:01:39.453701 12892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1899) took 1983 ms (client timeout 2223 ms). Trace:
I20260504 20:01:39.310755 12715 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:39.346436 15386 raft_consensus.cc:740] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:39.438045 15411 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:39.383443 15414 consensus_queue.cc:1059] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.124s
W20260504 20:01:39.454691 12896 rpcz_store.cc:269] 0504 20:01:37.134289 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.294085 (+159796us) service_pool.cc:224] Handling call
0504 20:01:39.409327 (+2115242us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b54001fb3d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:39.409499 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4404526336}
I20260504 20:01:39.382906 15449 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.431998 12790 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:39.459830 12893 rpcz_store.cc:269] 0504 20:01:37.564193 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.317544 (+1753351us) service_pool.cc:224] Handling call
0504 20:01:39.458022 (+140478us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.496775 13528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 402) took 1976 ms. Trace:
W20260504 20:01:39.541533 12892 rpcz_store.cc:269] 0504 20:01:37.470704 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.312138 (+1841434us) service_pool.cc:224] Handling call
0504 20:01:39.453637 (+141499us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:39.485076 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:39.516682 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.466583 12895 raft_consensus.cc:2410] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:39.465370 12898 raft_consensus.cc:2410] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260504 20:01:39.491655 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:39.481173 15345 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:39.618552 15372 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.494441 15415 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.513258 15025 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.464972 15390 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:39.470085 12897 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1898) took 2003 ms (client timeout 2200 ms). Trace:
W20260504 20:01:39.560235 13533 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 602) took 1824 ms (client timeout 1585 ms). Trace:
I20260504 20:01:39.618695 12791 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:39.474488 12716 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:39.453416 15364 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:39.491503 12891 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1900) took 2005 ms (client timeout 1922 ms). Trace:
W20260504 20:01:39.459910 12747 rpcz_store.cc:269] 0504 20:01:38.032557 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.304221 (+1271664us) service_pool.cc:224] Handling call
0504 20:01:39.441089 (+136868us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.619859 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1663) took 1968 ms. Trace:
W20260504 20:01:39.627029 12898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1904) took 2059 ms (client timeout 2168 ms). Trace:
W20260504 20:01:39.497602 12749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1797) took 3427 ms (client timeout 2113 ms). Trace:
I20260504 20:01:39.617427 13528 rpcz_store.cc:276] 0504 20:01:37.520865 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.521200 (+ 335us) service_pool.cc:224] Handling call
0504 20:01:37.525586 (+ 4386us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:39.347870 (+1822284us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.496306 (+148436us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b540068d154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:39.496657 (+ 351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4188199168}
I20260504 20:01:39.490321 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1664) took 1816 ms. Trace:
W20260504 20:01:39.509405 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 244692us
I20260504 20:01:39.369686 15454 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.470842 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:39.481561 15439 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:39.441015 12748 rpcz_store.cc:276] 0504 20:01:37.636409 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.636613 (+ 204us) service_pool.cc:224] Handling call
0504 20:01:37.679020 (+ 42407us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:37.701338 (+ 22318us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:37.701368 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:37.709930 (+ 8562us) log.cc:844] Serialized 26 byte log entry
0504 20:01:38.348183 (+638253us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b18004ab240. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d471 00007f015c969def
0504 20:01:39.320724 (+972541us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:39.320812 (+ 88us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.321017 (+ 205us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:39.325932 (+ 4915us) raft_consensus.cc:1685] finished
0504 20:01:39.325961 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:39.370219 (+ 44258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21877,"raft.queue_time_us":22373,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":18027,"spinlock_wait_cycles":13702016,"thread_start_us":972100,"threads_started":1,"wal-append.queue_time_us":1541579}
W20260504 20:01:39.630019 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:39.472262 12750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1680) took 1313 ms (client timeout 1548 ms). Trace:
W20260504 20:01:39.627611 12898 rpcz_store.cc:269] 0504 20:01:37.567564 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.375948 (+1808384us) service_pool.cc:224] Handling call
0504 20:01:39.626933 (+250985us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:39.627105 12891 rpcz_store.cc:269] 0504 20:01:37.486063 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.699623 (+213560us) service_pool.cc:224] Handling call
0504 20:01:39.491135 (+1791512us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54001b8d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:39.491382 (+ 247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3506281728}
I20260504 20:01:39.514801 15009 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260504 20:01:39.625259 13533 rpcz_store.cc:269] 0504 20:01:37.628934 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.630997 (+ 2063us) service_pool.cc:224] Handling call
0504 20:01:38.379429 (+748432us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b5400440320. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9261a0 00007f015c9b21f8 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006
0504 20:01:39.274026 (+894597us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b58004d4ed0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c90a48a 00007f015c98c827 00007f015c9a53c7 00007f015c9a6bbe 00007f015c9b265c 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c
0504 20:01:39.453341 (+179315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1243388416}
W20260504 20:01:39.625072 12897 rpcz_store.cc:269] 0504 20:01:37.467124 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.047237 (+1580113us) service_pool.cc:224] Handling call
0504 20:01:39.459337 (+412100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.629958 12745 rpcz_store.cc:276] 0504 20:01:37.673425 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.675816 (+ 2391us) service_pool.cc:224] Handling call
0504 20:01:37.685776 (+ 9960us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:39.363858 (+1678082us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.489830 (+125972us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b5400012a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:39.490235 (+ 405us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3854222720}
I20260504 20:01:39.488582 15373 raft_consensus.cc:2272] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Raft consensus is shut down!
W20260504 20:01:39.524245 15423 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.008s sys 0.030s
W20260504 20:01:39.627980 12749 rpcz_store.cc:269] 0504 20:01:36.069697 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.317455 (+247758us) service_pool.cc:224] Handling call
0504 20:01:39.497270 (+3179815us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b54003dac54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:39.497476 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5419224320}
I20260504 20:01:39.472688 15401 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:39.636392 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 177427us
I20260504 20:01:39.637044 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.639063 15327 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260504 20:01:39.633210 12750 rpcz_store.cc:269] 0504 20:01:38.158852 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:38.876189 (+717337us) service_pool.cc:224] Handling call
0504 20:01:39.457515 (+581326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.467037 12824 raft_consensus.cc:606] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:39.486969 15477 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:39.468799 15435 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.441040 12744 rpcz_store.cc:276] 0504 20:01:37.617609 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.617723 (+ 114us) service_pool.cc:224] Handling call
0504 20:01:37.618062 (+ 339us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:37.667217 (+ 49155us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:37.667255 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:37.667795 (+ 540us) log.cc:844] Serialized 26 byte log entry
0504 20:01:38.931947 (+1264152us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:39.290041 (+358094us) log.cc:844] Serialized 14 byte log entry
0504 20:01:39.295524 (+ 5483us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.295683 (+ 159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:39.295765 (+ 82us) raft_consensus.cc:1685] finished
0504 20:01:39.295786 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:39.296142 (+ 356us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48656,"prepare.queue_time_us":377980,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":71,"raft.queue_time_us":62039,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":10737,"thread_start_us":1273755,"threads_started":2,"wal-append.queue_time_us":1343884}
I20260504 20:01:39.644147 12743 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:39.477072 15441 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.627465 12746 rpcz_store.cc:276] 0504 20:01:37.651887 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.670574 (+ 18687us) service_pool.cc:224] Handling call
0504 20:01:37.670876 (+ 302us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:37.707984 (+ 37108us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:37.708022 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:37.708597 (+ 575us) log.cc:844] Serialized 26 byte log entry
0504 20:01:39.465165 (+1756568us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:39.465696 (+ 531us) log.cc:844] Serialized 14 byte log entry
0504 20:01:39.619450 (+153754us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.619558 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:39.619623 (+ 65us) raft_consensus.cc:1685] finished
0504 20:01:39.619634 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:39.619795 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":186214,"prepare.queue_time_us":317,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":201621,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":7088,"thread_start_us":1691277,"threads_started":1,"wal-append.queue_time_us":1308149}
I20260504 20:01:39.476150 15461 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:39.645988 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 172365us
I20260504 20:01:39.645826 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:39.661702 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:39.652760 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 221091us
I20260504 20:01:39.653105 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1656) took 2035 ms. Trace:
W20260504 20:01:39.442348 13525 rpcz_store.cc:269] 0504 20:01:37.065797 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.077482 (+ 11685us) service_pool.cc:224] Handling call
0504 20:01:38.296444 (+1218962us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3a3e 000000000035a3c9 00007f016190b8e0 00007f01619d4784 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:39.317184 (+1020740us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8225024}
I20260504 20:01:39.653319 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:39.645229 15423 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.198s user 0.012s sys 0.036s
I20260504 20:01:39.637476 15009 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:39.650442 15327 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:39.639058 15372 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.658453 15317 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:01:39.675115 12741 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:39.471908 15246 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.664567 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.470695 15179 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.658268 12899 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
W20260504 20:01:39.677047 12899 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1891) took 2269 ms (client timeout 2450 ms). Trace:
I20260504 20:01:39.675036 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:39.677627 12899 rpcz_store.cc:269] 0504 20:01:37.407721 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.506794 (+2099073us) service_pool.cc:224] Handling call
0504 20:01:39.676945 (+170151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.487952 15467 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.668823 12748 raft_consensus.cc:2393] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:39.470971 15434 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.670270 12894 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:39.671907 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:39.678594 12892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1697) took 1398 ms (client timeout 1673 ms). Trace:
W20260504 20:01:39.661044 12743 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1679) took 1623 ms (client timeout 1690 ms). Trace:
I20260504 20:01:39.470458 15483 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.678128 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:39.691694 12892 rpcz_store.cc:269] 0504 20:01:38.275288 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.635517 (+1360229us) service_pool.cc:224] Handling call
0504 20:01:39.673622 (+ 38105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.680799 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.682773 12893 raft_consensus.cc:2368] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5.
I20260504 20:01:39.674645 15327 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:39.684576 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.683400 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:39.684165 12894 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1905) took 2109 ms (client timeout 2191 ms). Trace:
I20260504 20:01:39.693326 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:39.687165 12714 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:39.672861 15009 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:39.678802 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.687354 12865 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:37318 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1895)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1906)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1709)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1712)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1713)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1714)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 438)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1862)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1864)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 428)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 430)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1877)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1896)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1717)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 432)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 433)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 434)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 435)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 436)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 437)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1722)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1724)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1725)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1726)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1728)
I20260504 20:01:39.690897 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:39.661823 15364 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:39.698714 12864 leader_election.cc:336] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009): Timed out: RequestConsensusVote RPC to 127.11.68.4:41009 timed out after 1.572s (SENT)
I20260504 20:01:39.692927 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:39.673722 15319 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:39.662899 12742 rpcz_store.cc:276] 0504 20:01:37.617928 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.618409 (+ 481us) service_pool.cc:224] Handling call
0504 20:01:37.618756 (+ 347us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:37.667382 (+ 48626us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:37.667426 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:37.680716 (+ 13290us) log.cc:844] Serialized 26 byte log entry
0504 20:01:39.300477 (+1619761us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:39.332145 (+ 31668us) log.cc:844] Serialized 14 byte log entry
0504 20:01:39.625441 (+293296us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.637845 (+ 12404us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:39.637903 (+ 58us) raft_consensus.cc:1685] finished
0504 20:01:39.637913 (+ 10us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:39.653011 (+ 15098us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70707,"prepare.queue_time_us":9717,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":82608,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":27807,"thread_start_us":1562631,"threads_started":2,"wal-append.queue_time_us":1383956}
W20260504 20:01:39.692406 12743 rpcz_store.cc:269] 0504 20:01:38.110355 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.274260 (+1163905us) service_pool.cc:224] Handling call
0504 20:01:39.657343 (+383083us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:39.686174 12748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1668) took 1673 ms (client timeout 1916 ms). Trace:
I20260504 20:01:39.671415 15439 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:39.706007 12748 rpcz_store.cc:269] 0504 20:01:38.011255 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.642368 (+1631113us) service_pool.cc:224] Handling call
0504 20:01:39.684560 (+ 42192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.695899 12791 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:39.684456 15417 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260504 20:01:39.707840 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:39.698130 12894 rpcz_store.cc:269] 0504 20:01:37.575058 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.451614 (+1876556us) service_pool.cc:224] Handling call
0504 20:01:39.684088 (+232474us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.702767 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.703956 12744 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:39.691056 15435 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:39.704311 12893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1698) took 1424 ms (client timeout 1732 ms). Trace:
W20260504 20:01:39.709184 12893 rpcz_store.cc:269] 0504 20:01:38.275638 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.631123 (+1355485us) service_pool.cc:224] Handling call
0504 20:01:39.700006 (+ 68883us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.704617 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1672) took 1673 ms. Trace:
I20260504 20:01:39.710433 12745 rpcz_store.cc:276] 0504 20:01:38.031494 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.644728 (+1613234us) service_pool.cc:224] Handling call
0504 20:01:39.704542 (+ 59814us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.706418 13528 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:39.666808 15345 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.686793 15317 raft_consensus.cc:697] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:39.711794 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:39.710150 12864 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:39.714478 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1677) took 1681 ms. Trace:
I20260504 20:01:39.715003 12746 rpcz_store.cc:276] 0504 20:01:38.033219 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.664220 (+1631001us) service_pool.cc:224] Handling call
0504 20:01:39.714402 (+ 50182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.708094 15417 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:39.717423 12893 raft_consensus.cc:2368] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
W20260504 20:01:39.715998 12899 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1702) took 1439 ms (client timeout 1844 ms). Trace:
W20260504 20:01:39.701596 13497 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.991s (SENT) [suppressed 40 similar messages]
W20260504 20:01:39.718079 12899 rpcz_store.cc:269] 0504 20:01:38.276867 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.683188 (+1406321us) service_pool.cc:224] Handling call
0504 20:01:39.715901 (+ 32713us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.710071 12896 raft_consensus.cc:2376] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5. Re-sending same reply.
W20260504 20:01:39.720908 12896 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1699) took 1445 ms (client timeout 1734 ms). Trace:
W20260504 20:01:39.721391 12896 rpcz_store.cc:269] 0504 20:01:38.275939 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.627155 (+1351216us) service_pool.cc:224] Handling call
0504 20:01:39.720842 (+ 93687us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.721745 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:39.724061 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:39.725242 15405 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:39.726003 15405 raft_consensus.cc:697] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:39.726454 15345 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:39.727308 15405 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.728003 15317 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.729900 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1906) took 2142 ms. Trace:
I20260504 20:01:39.720440 12748 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260504 20:01:39.730392 12893 rpcz_store.cc:276] 0504 20:01:37.578045 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.709960 (+2131915us) service_pool.cc:224] Handling call
0504 20:01:39.720624 (+ 10664us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.731221 12892 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:39.731997 15501 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:39.732558 15501 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:39.732761 12747 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
I20260504 20:01:39.732978 15501 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:39.733428 12747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1674) took 1701 ms (client timeout 1785 ms). Trace:
W20260504 20:01:39.733855 12747 rpcz_store.cc:269] 0504 20:01:38.032220 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.649910 (+1617690us) service_pool.cc:224] Handling call
0504 20:01:39.733334 (+ 83424us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.733117 12898 raft_consensus.cc:2393] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:39.735500 12891 raft_consensus.cc:2410] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:39.732347 12824 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
W20260504 20:01:39.737293 12898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1700) took 1461 ms (client timeout 1779 ms). Trace:
W20260504 20:01:39.737666 12891 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1846) took 3392 ms (client timeout 3711 ms). Trace:
I20260504 20:01:39.711648 15390 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:39.736119 12749 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:39.773821 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:39.723306 12897 raft_consensus.cc:2376] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6. Re-sending same reply.
W20260504 20:01:39.822008 12892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1895) took 2278 ms (client timeout 2773 ms). Trace:
W20260504 20:01:39.818305 12749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1670) took 1787 ms (client timeout 2188 ms). Trace:
I20260504 20:01:39.696687 15411 consensus_queue.cc:1048] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260504 20:01:39.774063 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.720510 15417 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:39.826372 12746 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:39.737876 12898 rpcz_store.cc:269] 0504 20:01:38.276224 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.637768 (+1361544us) service_pool.cc:224] Handling call
0504 20:01:39.737200 (+ 99432us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.825029 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:39.841079 12749 rpcz_store.cc:269] 0504 20:01:38.030711 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.651859 (+1621148us) service_pool.cc:224] Handling call
0504 20:01:39.818201 (+166342us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:39.824257 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 513981us
W20260504 20:01:39.825495 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 158194us
W20260504 20:01:39.824532 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 446557us
I20260504 20:01:39.774585 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:39.824000 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 125319us
W20260504 20:01:39.824793 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 100046us
W20260504 20:01:39.738174 12891 rpcz_store.cc:269] 0504 20:01:36.345582 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.649012 (+3303430us) service_pool.cc:224] Handling call
0504 20:01:39.737594 (+ 88582us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.443662 12646 rpcz_store.cc:276] 0504 20:01:29.660070 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:29.679906 (+ 19836us) service_pool.cc:224] Handling call
0504 20:01:39.221160 (+9541254us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:34.010166 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:34.011037 (+ 871us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:34.011090 (+ 53us) write_op.cc:435] Acquired schema lock
0504 20:01:34.011111 (+ 21us) tablet.cc:662] Decoding operations
0504 20:01:34.147761 (+136650us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:34.147830 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:34.147868 (+ 38us) tablet.cc:685] Acquiring locks for 14 operations
0504 20:01:34.164795 (+ 16927us) tablet.cc:701] Row locks acquired
0504 20:01:34.164817 (+ 22us) write_op.cc:260] PREPARE: finished
0504 20:01:34.164988 (+ 171us) write_op.cc:270] Start()
0504 20:01:34.165191 (+ 203us) write_op.cc:276] Timestamp: P: 1777924894164937 usec, L: 0
0504 20:01:34.165231 (+ 40us) op_driver.cc:348] REPLICATION: starting
0504 20:01:34.166431 (+ 1200us) log.cc:844] Serialized 5483 byte log entry
0504 20:01:35.882142 (+1715711us) op_driver.cc:464] REPLICATION: finished
0504 20:01:36.631236 (+749094us) write_op.cc:301] APPLY: starting
0504 20:01:36.633990 (+ 2754us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:36.735245 (+101255us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:36.735413 (+ 168us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:36.784378 (+ 48965us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:36.784770 (+ 392us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0504 20:01:36.784841 (+ 71us) write_op.cc:312] APPLY: finished
0504 20:01:36.785342 (+ 501us) log.cc:844] Serialized 101 byte log entry
0504 20:01:37.167080 (+381738us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:37.196707 (+ 29627us) write_op.cc:454] Released schema lock
0504 20:01:37.203184 (+ 6477us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:37.219185 (+ 16001us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:37.219221 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24040,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":613029,"apply.run_cpu_time_us":45872,"apply.run_wall_time_us":680199,"num_ops":14,"prepare.queue_time_us":510648,"prepare.run_cpu_time_us":70316,"prepare.run_wall_time_us":582859,"raft.queue_time_us":778141,"raft.run_cpu_time_us":21821,"raft.run_wall_time_us":383083,"replication_time_us":1716700,"thread_start_us":1573315,"threads_started":5,"wal-append.queue_time_us":950489,"wal-append.run_cpu_time_us":13474,"wal-append.run_wall_time_us":2946145}]]}
I20260504 20:01:39.778148 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1671) took 1704 ms. Trace:
I20260504 20:01:39.873414 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.881392 12748 rpcz_store.cc:276] 0504 20:01:38.031094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.710251 (+1679157us) service_pool.cc:224] Handling call
0504 20:01:39.717218 (+ 6967us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:39.734863 (+ 17645us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.735126 (+ 263us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:39.782678 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 47235.8us
I20260504 20:01:39.828002 12742 raft_consensus.cc:1240] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:39.883611 12896 raft_consensus.cc:2368] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6.
W20260504 20:01:39.824182 15485 log.cc:927] Time spent T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.165s user 0.000s sys 0.000s
I20260504 20:01:39.888868 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1673) took 1852 ms. Trace:
W20260504 20:01:39.860661 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 82733.3us
I20260504 20:01:39.773489 15246 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:39.826507 12824 raft_consensus.cc:606] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:39.860419 13528 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:39.822963 12892 rpcz_store.cc:269] 0504 20:01:37.453537 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.695657 (+2242120us) service_pool.cc:224] Handling call
0504 20:01:39.731906 (+ 36249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.730311 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:39.822777 12897 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1701) took 1546 ms (client timeout 1838 ms). Trace:
W20260504 20:01:39.896497 12897 rpcz_store.cc:269] 0504 20:01:38.276559 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.660376 (+1383817us) service_pool.cc:224] Handling call
0504 20:01:39.822705 (+162329us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.773556 15439 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.889881 12899 raft_consensus.cc:2393] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:39.889761 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1704) took 1183 ms. Trace:
I20260504 20:01:39.901563 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:39.902058 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:39.903191 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.904820 12892 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:39.905555 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1709) took 1197 ms. Trace:
I20260504 20:01:39.906039 12892 rpcz_store.cc:276] 0504 20:01:38.707820 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.902430 (+1194610us) service_pool.cc:224] Handling call
0504 20:01:39.905457 (+ 3027us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.906905 12790 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:39.907517 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:39.908450 15488 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:39.908633 12892 raft_consensus.cc:2376] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5. Re-sending same reply.
I20260504 20:01:39.909070 15488 raft_consensus.cc:697] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:39.775230 12745 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:39.889338 12742 rpcz_store.cc:276] 0504 20:01:38.031880 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.708950 (+1677070us) service_pool.cc:224] Handling call
0504 20:01:39.816339 (+107389us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:39.883972 (+ 67633us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.884201 (+ 229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.910367 15488 consensus_queue.cc:237] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.891688 12893 raft_consensus.cc:2376] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2. Re-sending same reply.
I20260504 20:01:39.911770 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1706) took 1205 ms. Trace:
I20260504 20:01:39.777962 15295 consensus_queue.cc:1059] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.135s
I20260504 20:01:39.912392 12893 rpcz_store.cc:276] 0504 20:01:38.706768 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.738510 (+1031742us) service_pool.cc:224] Handling call
0504 20:01:39.911704 (+173194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.781560 15373 ts_tablet_manager.cc:1916] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260504 20:01:39.892756 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:39.893271 15116 consensus_meta.cc:287] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.796s user 0.011s sys 0.011s
I20260504 20:01:39.914894 15116 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:39.892086 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.909535 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1711) took 1201 ms. Trace:
I20260504 20:01:39.825994 15396 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:39.910945 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 366) took 1032 ms. Trace:
I20260504 20:01:39.901345 12896 rpcz_store.cc:276] 0504 20:01:38.706119 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.737256 (+1031137us) service_pool.cc:224] Handling call
0504 20:01:39.889666 (+152410us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.917179 15332 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:39.887194 12716 leader_election.cc:336] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.584s (SENT)
I20260504 20:01:39.918344 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1707) took 1211 ms. Trace:
I20260504 20:01:39.913553 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.918560 12892 rpcz_store.cc:276] 0504 20:01:38.708554 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.906943 (+1198389us) service_pool.cc:224] Handling call
0504 20:01:39.909436 (+ 2493us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:39.772698 15503 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20260504 20:01:39.883962 12750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1678) took 1850 ms. Trace:
I20260504 20:01:39.924273 12899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1705) took 1217 ms. Trace:
W20260504 20:01:39.931350 13498 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:39.925374 12745 rpcz_store.cc:276] 0504 20:01:38.878625 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.723174 (+844549us) service_pool.cc:224] Handling call
0504 20:01:39.774724 (+ 51550us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:39.910496 (+135772us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.910849 (+ 353us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.934330 12899 rpcz_store.cc:276] 0504 20:01:38.706457 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.737481 (+1031024us) service_pool.cc:224] Handling call
0504 20:01:39.924191 (+186710us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.933772 12750 rpcz_store.cc:276] 0504 20:01:38.033547 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.705284 (+1671737us) service_pool.cc:224] Handling call
0504 20:01:39.883861 (+178577us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.935115 12898 rpcz_store.cc:276] 0504 20:01:38.707109 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.861161 (+1154052us) service_pool.cc:224] Handling call
0504 20:01:39.918215 (+ 57054us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.942619 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:39.947573 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:39.949458 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.941826 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:39.924438 15467 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:39.949359 12715 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:39.952313 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 65187.1us
I20260504 20:01:39.925628 15116 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:39.955051 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:39.960685 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.960808 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.968652 12896 raft_consensus.cc:2368] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7.
I20260504 20:01:39.971050 15495 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:39.976787 15454 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:39.977000 15495 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:39.977475 15495 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:39.782167 15386 consensus_queue.cc:260] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:39.973591 12899 raft_consensus.cc:2393] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:39.985137 12899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1714) took 1275 ms. Trace:
I20260504 20:01:39.985549 12899 rpcz_store.cc:276] 0504 20:01:38.709490 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.935138 (+1225648us) service_pool.cc:224] Handling call
0504 20:01:39.985070 (+ 49932us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.986912 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1715) took 1115 ms. Trace:
I20260504 20:01:39.964898 12891 raft_consensus.cc:2376] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5. Re-sending same reply.
I20260504 20:01:39.984763 12893 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
I20260504 20:01:39.987793 12896 rpcz_store.cc:276] 0504 20:01:38.871142 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.935682 (+1064540us) service_pool.cc:224] Handling call
0504 20:01:39.986796 (+ 51114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.990796 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1712) took 1282 ms. Trace:
I20260504 20:01:39.990329 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1708) took 1282 ms. Trace:
I20260504 20:01:39.991173 12893 rpcz_store.cc:276] 0504 20:01:38.708847 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.913170 (+1204323us) service_pool.cc:224] Handling call
0504 20:01:39.990734 (+ 77564us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.991338 12891 rpcz_store.cc:276] 0504 20:01:38.707514 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.881728 (+1174214us) service_pool.cc:224] Handling call
0504 20:01:39.990243 (+108515us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:39.982856 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:39.989149 12824 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:39.984541 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:39.969705 12897 raft_consensus.cc:2376] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7. Re-sending same reply.
I20260504 20:01:39.998176 12824 raft_consensus.cc:606] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:39.998920 12824 raft_consensus.cc:614] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:39.998198 13524 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:39.998929 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1710) took 1290 ms. Trace:
I20260504 20:01:40.001864 12897 rpcz_store.cc:276] 0504 20:01:38.708217 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.924758 (+1216541us) service_pool.cc:224] Handling call
0504 20:01:39.998828 (+ 74070us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.001338 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1713) took 1292 ms. Trace:
I20260504 20:01:40.004366 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:39.966833 15477 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260504 20:01:40.003162 12892 rpcz_store.cc:276] 0504 20:01:38.709190 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.938603 (+1229413us) service_pool.cc:224] Handling call
0504 20:01:40.001257 (+ 62654us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.014755 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:40.016279 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:40.000842 15434 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:40.023104 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:40.023965 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 140717us
I20260504 20:01:40.026327 12790 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:40.031213 12824 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260504 20:01:40.024169 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.022053 13525 raft_consensus.cc:1240] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:40.034097 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260504 20:01:40.042246 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.043932 15405 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:40.044394 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.045004 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:01:40.021384 15411 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20260504 20:01:40.045933 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.029629 15432 consensus_queue.cc:1048] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260504 20:01:40.040601 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.040469 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.041347 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.043972 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.044498 15405 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:40.051157 12823 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 5
I20260504 20:01:40.050590 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:40.057412 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:40.057041 12897 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:40.062247 12821 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:40.075696 15478 raft_consensus.cc:3055] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:40.076295 15478 raft_consensus.cc:740] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:40.077364 15386 raft_consensus.cc:3060] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:40.054595 15405 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:40.114694 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 80037.7us
W20260504 20:01:40.115428 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 43912.7us
I20260504 20:01:40.117970 12744 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
W20260504 20:01:40.118178 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.118680 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57934 (request call id 365) took 1240 ms. Trace:
W20260504 20:01:40.120096 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:40.120580 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.122529 12892 raft_consensus.cc:2393] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:40.123490 15295 consensus_queue.cc:1048] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:40.126266 12822 raft_consensus.cc:2410] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260504 20:01:40.125975 15509 consensus_queue.cc:1059] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260504 20:01:40.128717 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:40.129436 12894 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:40.130160 12893 raft_consensus.cc:2393] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:40.131372 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1893) took 2687 ms. Trace:
I20260504 20:01:40.036502 15483 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:40.131752 12893 rpcz_store.cc:276] 0504 20:01:37.443690 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.009844 (+2566154us) service_pool.cc:224] Handling call
0504 20:01:40.131305 (+121461us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:40.132601 15508 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20260504 20:01:40.133040 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:40.134121 15319 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:40.134639 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:40.136206 12893 raft_consensus.cc:1240] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:40.136714 12896 raft_consensus.cc:1275] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:40.136826 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 428) took 3068 ms. Trace:
I20260504 20:01:40.131387 13499 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:40.138733 12894 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1718) took 1177 ms (client timeout 1511 ms). Trace:
I20260504 20:01:40.139156 15417 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.139840 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1669) took 2109 ms. Trace:
I20260504 20:01:40.137248 12893 rpcz_store.cc:276] 0504 20:01:37.068031 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.135606 (+3067575us) service_pool.cc:224] Handling call
0504 20:01:40.136011 (+ 405us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.136586 (+ 575us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.136752 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.118964 12744 rpcz_store.cc:276] 0504 20:01:38.878211 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.666902 (+788691us) service_pool.cc:224] Handling call
0504 20:01:40.118616 (+451714us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.140654 12743 rpcz_store.cc:276] 0504 20:01:38.030182 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.705972 (+1675790us) service_pool.cc:224] Handling call
0504 20:01:39.730932 (+ 24960us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:39.828884 (+ 97952us) raft_consensus.cc:1534] Early marking committed up to index 2
0504 20:01:39.873728 (+ 44844us) log.cc:844] Serialized 14 byte log entry
0504 20:01:39.937693 (+ 63965us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:39.950747 (+ 13054us) log.cc:844] Serialized 26 byte log entry
0504 20:01:39.950958 (+ 211us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:39.951371 (+ 413us) log.cc:844] Serialized 14 byte log entry
0504 20:01:39.957210 (+ 5839us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.957298 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:40.137306 (+180008us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b180046ea00. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598debe6 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159910a33 00007f0159912e01
0504 20:01:40.138635 (+ 1329us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b180046ea00. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159910a33 00007f0159912e01 00007f0159912db9 00007f0159912d49 00007f0159912d11 00007f015991200d 0000000000379254 0000000000379099 00007f01598f3fba
0504 20:01:40.139580 (+ 945us) raft_consensus.cc:1685] finished
0504 20:01:40.139608 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:40.139749 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22139,"prepare.queue_time_us":212238,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":74,"raft.queue_time_us":8837,"raft.run_cpu_time_us":453,"raft.run_wall_time_us":24278,"spinlock_wait_cycles":10630400,"thread_start_us":48050,"threads_started":1,"wal-append.queue_time_us":112020}
I20260504 20:01:40.143236 15009 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:40.143482 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 430) took 2979 ms. Trace:
I20260504 20:01:40.144699 12891 raft_consensus.cc:2368] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 8.
I20260504 20:01:40.146884 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1911) took 2410 ms. Trace:
I20260504 20:01:40.144143 12893 rpcz_store.cc:276] 0504 20:01:37.163909 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.141538 (+2977629us) service_pool.cc:224] Handling call
0504 20:01:40.142325 (+ 787us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.142857 (+ 532us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:40.142923 (+ 66us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:40.143026 (+ 103us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:40.143093 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.143195 (+ 102us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:40.143392 (+ 197us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:40.149946 12716 leader_election.cc:336] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.922s (SENT)
I20260504 20:01:40.150578 12893 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:40.147912 12825 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260504 20:01:40.147318 12818 rpcz_store.cc:276] 0504 20:01:37.736118 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:37.736291 (+ 173us) service_pool.cc:224] Handling call
0504 20:01:37.736661 (+ 370us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:38.893817 (+1157156us) raft_consensus.cc:1534] Early marking committed up to index 3
0504 20:01:39.281414 (+387597us) log.cc:844] Serialized 14 byte log entry
0504 20:01:39.522472 (+241058us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:39.523659 (+ 1187us) log.cc:844] Serialized 26 byte log entry
0504 20:01:39.523838 (+ 179us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:39.523877 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:39.523983 (+ 106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:40.146478 (+622495us) raft_consensus.cc:1685] finished
0504 20:01:40.146504 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:40.146679 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24572,"prepare.queue_time_us":455191,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":82,"raft.queue_time_us":110148,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":16138,"thread_start_us":424511,"threads_started":2,"wal-append.queue_time_us":370501}
I20260504 20:01:40.147178 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1716) took 1269 ms. Trace:
W20260504 20:01:40.151628 12716 leader_election.cc:336] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.058s (SENT)
I20260504 20:01:40.153581 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1881) took 1167 ms. Trace:
I20260504 20:01:40.154449 12891 rpcz_store.cc:276] 0504 20:01:38.877976 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.997027 (+1119051us) service_pool.cc:224] Handling call
0504 20:01:40.146594 (+149567us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54003ba454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:40.146998 (+ 404us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277469568}
I20260504 20:01:40.155786 12825 rpcz_store.cc:276] 0504 20:01:38.986701 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:38.998513 (+ 11812us) service_pool.cc:224] Handling call
0504 20:01:40.147444 (+1148931us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.152332 (+ 4888us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.153363 (+ 1031us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b540014dcd0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:40.153510 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2183676160}
I20260504 20:01:40.157282 12741 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:40.151683 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1871) took 2910 ms. Trace:
I20260504 20:01:40.158267 12891 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:40.155043 12716 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:40.160270 15025 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:40.160660 15390 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:40.145561 15009 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:40.163929 15364 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:40.165426 15401 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.508s user 0.009s sys 0.003s
I20260504 20:01:40.165565 15465 raft_consensus.cc:3055] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:40.166155 15465 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:40.077423 15478 consensus_queue.cc:260] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.167573 12715 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:40.167363 15465 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.168895 15465 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:40.169090 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 427) took 3101 ms. Trace:
I20260504 20:01:40.169518 12897 rpcz_store.cc:276] 0504 20:01:37.067708 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.061755 (+2994047us) service_pool.cc:224] Handling call
0504 20:01:40.168437 (+106682us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.168672 (+ 235us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:40.168698 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:40.168771 (+ 73us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:40.168816 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.168879 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:40.169021 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.170615 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:40.160027 12741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1676) took 2127 ms (client timeout 1779 ms). Trace:
W20260504 20:01:40.172111 12741 rpcz_store.cc:269] 0504 20:01:38.032879 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.470417 (+1437538us) service_pool.cc:224] Handling call
0504 20:01:40.159879 (+689462us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.158916 12893 rpcz_store.cc:276] 0504 20:01:37.241856 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.149438 (+2907582us) service_pool.cc:224] Handling call
0504 20:01:40.150047 (+ 609us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.151220 (+ 1173us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.151575 (+ 355us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.130013 15501 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:40.139344 12894 rpcz_store.cc:269] 0504 20:01:38.961055 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.719932 (+758877us) service_pool.cc:224] Handling call
0504 20:01:40.138644 (+418712us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.137408 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1864) took 3106 ms. Trace:
I20260504 20:01:40.123154 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1719) took 1146 ms. Trace:
I20260504 20:01:40.186523 12821 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:40.189270 12896 rpcz_store.cc:276] 0504 20:01:37.031470 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.049714 (+3018244us) service_pool.cc:224] Handling call
0504 20:01:40.136317 (+ 86603us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.137123 (+ 806us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.137334 (+ 211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.189481 12892 rpcz_store.cc:276] 0504 20:01:38.976542 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.014058 (+1037516us) service_pool.cc:224] Handling call
0504 20:01:40.123078 (+109020us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.178201 13528 raft_consensus.cc:2393] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:40.190950 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1877) took 2844 ms. Trace:
W20260504 20:01:40.193130 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 61809.5us
W20260504 20:01:40.194398 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:40.196130 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.153517 12898 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:40.157061 15327 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:40.199198 12897 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260504 20:01:40.202663 12897 pending_rounds.cc:85] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 3
I20260504 20:01:40.192736 12891 rpcz_store.cc:276] 0504 20:01:37.330195 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.156436 (+2826241us) service_pool.cc:224] Handling call
0504 20:01:40.157703 (+ 1267us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.173895 (+ 16192us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.174436 (+ 541us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.209075 15415 consensus_queue.cc:1059] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:40.210716 15415 raft_consensus.cc:3055] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:40.197563 15477 consensus_queue.cc:1059] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260504 20:01:40.211114 15415 raft_consensus.cc:740] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:40.211617 15477 consensus_queue.cc:1243] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer ef6057d9a49e4942b47ea0fcd66ab6f7 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260504 20:01:40.212208 15477 raft_consensus.cc:3055] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:40.212150 15415 consensus_queue.cc:260] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.212914 15477 raft_consensus.cc:740] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:40.213918 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:40.214771 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.215256 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:40.216089 13533 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:40.198119 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:40.206521 12897 pending_rounds.cc:107] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260504 20:01:40.224004 12897 raft_consensus.cc:2889] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:40.226228 12893 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260504 20:01:40.192725 15401 raft_consensus.cc:2804] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:40.166102 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:40.228586 12817 raft_consensus.cc:2393] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:40.206276 15345 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:40.238119 15345 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:40.238601 15345 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:40.242098 15179 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:40.247570 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:40.248224 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.248442 12890 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
W20260504 20:01:40.249125 12890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1892) took 2811 ms (client timeout 2350 ms). Trace:
I20260504 20:01:40.249374 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.250270 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.251039 12822 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:40.251950 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1848) took 3905 ms (client timeout 2099 ms). Trace:
W20260504 20:01:40.252461 12895 rpcz_store.cc:269] 0504 20:01:36.346438 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:36.361649 (+ 15211us) service_pool.cc:224] Handling call
0504 20:01:40.251672 (+3890023us) spinlock_profiling.cc:243] Waited 2.79 s on lock 0x7b54001fa9d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:40.251855 (+ 183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6422328064}
I20260504 20:01:40.254508 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:40.214075 15477 consensus_queue.cc:260] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.257318 15477 raft_consensus.cc:3060] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:40.257191 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.262576 12741 raft_consensus.cc:1240] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:40.263285 12750 raft_consensus.cc:1240] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:40.264874 15400 raft_consensus.cc:3055] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:40.265317 15400 raft_consensus.cc:740] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:40.266464 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:40.267401 12745 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:40.268556 15372 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:40.269964 15469 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.760s user 0.008s sys 0.007s
I20260504 20:01:40.270179 15415 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:40.270493 15469 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.272292 13532 raft_consensus.cc:2393] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 7: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
W20260504 20:01:40.229106 12716 leader_election.cc:336] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.105s (SENT)
W20260504 20:01:40.274096 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 50552us
W20260504 20:01:40.274785 12716 leader_election.cc:336] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.131s (SENT)
I20260504 20:01:40.275173 12716 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:40.276925 15246 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:40.281119 15417 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:40.280413 12647 catalog_manager.cc:5671] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 2 to 3, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
W20260504 20:01:40.249534 12890 rpcz_store.cc:269] 0504 20:01:37.437271 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.470790 (+2033519us) service_pool.cc:224] Handling call
0504 20:01:40.247417 (+776627us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3a3e 000000000035a3c9 00007f016190b8e0 00007f01619d4784 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:40.249020 (+ 1603us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5884288}
I20260504 20:01:40.234395 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1703) took 1949 ms. Trace:
I20260504 20:01:40.231310 12894 raft_consensus.cc:1275] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:40.252087 12816 raft_consensus.cc:1240] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:40.291923 12893 rpcz_store.cc:276] 0504 20:01:38.705544 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.175025 (+1469481us) service_pool.cc:224] Handling call
0504 20:01:40.225874 (+ 50849us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.226831 (+ 957us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.226997 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.292426 12894 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1717) took 1405 ms. Trace:
I20260504 20:01:40.249550 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.292789 12894 rpcz_store.cc:276] 0504 20:01:38.933117 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.189365 (+1256248us) service_pool.cc:224] Handling call
0504 20:01:40.205078 (+ 15713us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.292113 (+ 87035us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.292354 (+ 241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.235133 12891 raft_consensus.cc:1240] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:40.234196 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:40.291534 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.296669 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:40.297497 13528 raft_consensus.cc:2393] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:40.303349 15327 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:40.299937 13532 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40940 (request call id 333) took 1034 ms. Trace:
I20260504 20:01:40.299706 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1896) took 2843 ms. Trace:
I20260504 20:01:40.273051 15469 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:40.305954 12897 rpcz_store.cc:276] 0504 20:01:37.455965 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.170544 (+2714579us) service_pool.cc:224] Handling call
0504 20:01:40.170889 (+ 345us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.299495 (+128606us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.299655 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:40.306453 12745 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:01:40.306794 12892 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:40.304563 13532 rpcz_store.cc:276] 0504 20:01:39.281080 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.282249 (+ 1169us) service_pool.cc:224] Handling call
0504 20:01:40.299761 (+1017512us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b5400440a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:40.299882 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2272136064}
I20260504 20:01:40.314323 12825 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260504 20:01:40.314188 12716 leader_election.cc:336] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.191s (SENT)
I20260504 20:01:40.315701 12716 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:40.313040 12823 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:40.317593 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:40.317922 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:40.313867 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.318300 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:40.300925 15478 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:40.320700 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:40.322075 12896 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260504 20:01:40.325424 12824 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:40.326267 12898 raft_consensus.cc:1275] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260504 20:01:40.298076 15401 raft_consensus.cc:697] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:40.328027 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.329275 12893 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:40.329689 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1720) took 1335 ms. Trace:
I20260504 20:01:40.329794 12890 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:40.330159 12891 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260504 20:01:40.330998 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.326671 12898 pending_rounds.cc:85] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 5
I20260504 20:01:40.337555 12898 pending_rounds.cc:107] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260504 20:01:40.338001 12898 raft_consensus.cc:2889] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:40.332759 12897 raft_consensus.cc:1240] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:40.337951 12896 rpcz_store.cc:276] 0504 20:01:38.989576 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.191813 (+1202237us) service_pool.cc:224] Handling call
0504 20:01:40.312055 (+120242us) spinlock_profiling.cc:243] Waited 63.8 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:40.312257 (+ 202us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.324393 (+ 12136us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.324595 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":146842880}
I20260504 20:01:40.335762 15417 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:40.333762 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:40.331383 12647 catalog_manager.cc:5671] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 5 to 6, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 6 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:40.328282 13529 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260504 20:01:40.328500 15025 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:40.356808 12891 raft_consensus.cc:1275] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:40.358901 12865 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:40.359804 12894 raft_consensus.cc:1240] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:40.360693 12823 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:40.354022 15435 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:40.363766 15435 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:40.364682 15511 log.cc:927] Time spent T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260504 20:01:40.360944 12893 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260504 20:01:40.367623 13499 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:40.369321 12896 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260504 20:01:40.369861 12896 pending_rounds.cc:85] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 3
I20260504 20:01:40.373410 12899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1862) took 3647 ms. Trace:
I20260504 20:01:40.373981 12899 rpcz_store.cc:276] 0504 20:01:36.725748 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.029965 (+3304217us) service_pool.cc:224] Handling call
0504 20:01:40.146216 (+116251us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.146782 (+ 566us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:40.146818 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:40.147428 (+ 610us) log.cc:844] Serialized 26 byte log entry
0504 20:01:40.294892 (+147464us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:40.294932 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.295000 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:40.373123 (+ 78123us) raft_consensus.cc:1685] finished
0504 20:01:40.373148 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:40.373313 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5784,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":230,"thread_start_us":147009,"threads_started":1,"wal-append.queue_time_us":223185}
I20260504 20:01:40.380288 12744 raft_consensus.cc:1275] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260504 20:01:40.381762 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.382733 12742 raft_consensus.cc:2393] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:40.383191 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.383925 12825 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:40.384891 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:40.385294 12633 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 0e8eba4dbb8d436f81062ca3bbf46788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 5ff0ef72ec3b4667893a0420d3c7f923 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:40.386229 15372 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:40.386883 15372 raft_consensus.cc:697] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:40.386991 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.387930 15372 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:40.387045 15510 log.cc:927] Time spent T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260504 20:01:40.392354 15469 consensus_queue.cc:1059] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260504 20:01:40.393661 15423 raft_consensus.cc:3055] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:40.394183 15423 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:40.394282 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.395231 12748 raft_consensus.cc:2393] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:40.395120 15423 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.396229 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.396930 12748 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:40.397498 15435 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:40.397836 13499 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:40.398185 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:40.398901 12747 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:40.399184 13499 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:40.400969 15434 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:40.401098 13529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 405) took 1019 ms. Trace:
I20260504 20:01:40.401564 13529 rpcz_store.cc:276] 0504 20:01:39.381349 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.381535 (+ 186us) service_pool.cc:224] Handling call
0504 20:01:40.307560 (+926025us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.356942 (+ 49382us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.400886 (+ 43944us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b5400440a50. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:40.401027 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2110848256}
I20260504 20:01:40.402768 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:40.403422 15380 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:40.404160 15380 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:40.404601 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 66889.9us
I20260504 20:01:40.403483 15461 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:40.404641 15380 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:40.351428 15400 consensus_queue.cc:260] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.407199 15400 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:40.407420 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:40.408046 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 436) took 1028 ms. Trace:
I20260504 20:01:40.409197 15179 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:40.410303 12865 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:40.411692 12863 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:40.398118 15435 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:40.415022 15435 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:40.415645 15461 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:40.416437 15461 consensus_queue.cc:1059] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:40.356899 15390 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:40.417299 15386 consensus_queue.cc:1059] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:01:40.418107 15390 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:40.418630 15386 raft_consensus.cc:3055] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:40.419023 15386 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:40.420035 15386 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:40.420934 15390 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:40.421320 15386 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:40.422222 15501 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:40.423769 15009 raft_consensus.cc:3055] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:40.424325 15009 raft_consensus.cc:740] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:40.424727 12749 raft_consensus.cc:1275] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260504 20:01:40.414400 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 435) took 1097 ms. Trace:
I20260504 20:01:40.427160 12898 rpcz_store.cc:276] 0504 20:01:39.317245 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.220868 (+903623us) service_pool.cc:224] Handling call
0504 20:01:40.221351 (+ 483us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.352300 (+130949us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.414088 (+ 61788us) spinlock_profiling.cc:243] Waited 83.9 ms on lock 0x7b54002cc4d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:40.414313 (+ 225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192996096}
I20260504 20:01:40.403673 13532 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:40.431020 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.431846 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.432798 12745 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:40.434612 12896 pending_rounds.cc:107] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260504 20:01:40.435115 12896 raft_consensus.cc:2889] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:40.417495 15435 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:40.393529 12715 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:40.451018 12746 raft_consensus.cc:1275] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260504 20:01:40.452745 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 145217us
I20260504 20:01:40.453909 12822 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:40.441901 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.452905 12790 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:40.441653 12823 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:01:40.425520 15009 consensus_queue.cc:260] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:40.462152 15467 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:40.453441 12647 catalog_manager.cc:5671] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 4 to 5, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 5 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:40.408509 12895 rpcz_store.cc:276] 0504 20:01:39.379658 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.253321 (+873663us) service_pool.cc:224] Handling call
0504 20:01:40.253653 (+ 332us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:40.407009 (+153356us) raft_consensus.cc:1534] Early marking committed up to index 5
0504 20:01:40.407044 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:40.407130 (+ 86us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:40.407188 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.407782 (+ 594us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54000f4954. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:40.407832 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:40.407978 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10312,"raft.queue_time_us":12317,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"spinlock_wait_cycles":328044416}
I20260504 20:01:40.464393 12890 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
W20260504 20:01:40.472043 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.462694 13528 raft_consensus.cc:1240] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:40.426095 15401 consensus_queue.cc:237] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.477869 12745 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260504 20:01:40.478729 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.482352 12714 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:40.493427 15434 raft_consensus.cc:697] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
W20260504 20:01:40.499589 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 73305us
I20260504 20:01:40.505683 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:40.506999 12863 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:40.508543 12716 leader_election.cc:336] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.200s (SENT)
I20260504 20:01:40.509249 12716 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:40.505445 15327 consensus_queue.cc:1059] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260504 20:01:40.513999 12744 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:40.518775 12746 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 9: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:40.365780 15025 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:40.462169 15439 consensus_queue.cc:1048] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260504 20:01:40.477315 15467 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:40.534790 12745 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260504 20:01:40.480589 15179 raft_consensus.cc:2804] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:40.541246 12742 raft_consensus.cc:1240] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:40.535440 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 53245.8us
W20260504 20:01:40.545394 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 135228us
I20260504 20:01:40.510113 15009 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:40.585081 12714 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:40.491510 15405 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:40.589103 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:40.589593 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 127208us
W20260504 20:01:40.591322 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 197886us
I20260504 20:01:40.603221 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:40.581517 15495 consensus_meta.cc:287] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.603s user 0.013s sys 0.001s
I20260504 20:01:40.547180 15467 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:40.544251 15435 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:40.533413 15364 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:40.544860 15179 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:40.611845 12715 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:40.544719 15116 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:40.586961 15390 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:40.617441 12863 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:40.619545 13532 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:40.612183 15434 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:40.627014 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.627115 13499 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:40.629854 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 75383.5us
I20260504 20:01:40.630059 15411 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260504 20:01:40.630177 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 121709us
I20260504 20:01:40.630353 15414 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:40.630767 12865 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:40.631105 15414 raft_consensus.cc:697] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:40.631675 15423 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:40.632227 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 336) took 1156 ms. Trace:
W20260504 20:01:40.631170 12716 leader_election.cc:336] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.223s (SENT)
I20260504 20:01:40.632718 13527 rpcz_store.cc:276] 0504 20:01:39.475887 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:39.623591 (+147704us) service_pool.cc:224] Handling call
0504 20:01:40.254520 (+630929us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3a3e 000000000035a3c9 00007f016190b8e0 00007f01619d4784 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:40.254713 (+ 193us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.255173 (+ 460us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:40.255211 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:40.255880 (+ 669us) log.cc:844] Serialized 26 byte log entry
0504 20:01:40.466973 (+211093us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:40.610980 (+144007us) log.cc:844] Serialized 14 byte log entry
0504 20:01:40.623108 (+ 12128us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.631797 (+ 8689us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:40.631863 (+ 66us) raft_consensus.cc:1685] finished
0504 20:01:40.631972 (+ 109us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:40.632154 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34,"raft.queue_time_us":4815,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":6099,"spinlock_wait_cycles":9979648,"thread_start_us":262655,"threads_started":2,"wal-append.queue_time_us":190745}
I20260504 20:01:40.634577 12716 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:40.632333 15414 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.635815 15483 raft_consensus.cc:3055] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:40.608637 15246 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
W20260504 20:01:40.621074 12791 leader_election.cc:336] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.673s (SENT)
I20260504 20:01:40.650585 12791 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:40.616555 15364 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:40.647624 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:40.630604 15411 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:40.617740 15179 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:40.658773 12863 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:40.666882 12750 raft_consensus.cc:2393] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:40.679275 12896 pending_rounds.cc:85] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 5
I20260504 20:01:40.679682 12896 pending_rounds.cc:107] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260504 20:01:40.613843 15495 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.679931 12896 raft_consensus.cc:2889] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:40.646550 15483 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:40.681003 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 68046.5us
W20260504 20:01:40.684912 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 73160.7us
I20260504 20:01:40.665402 15364 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:40.670130 15411 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:40.700330 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 79307us
W20260504 20:01:40.704721 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.708889 12824 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:40.673028 15417 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:01:40.711032 12714 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:40.710076 12647 catalog_manager.cc:5671] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 3 to 4, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 4 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:40.678943 15465 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:40.719630 13531 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:01:40.719857 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.678707 15246 raft_consensus.cc:697] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:40.723358 15327 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:01:40.684617 15396 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:01:40.740466 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 109830us
I20260504 20:01:40.742864 15435 raft_consensus.cc:697] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:40.695066 15530 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
W20260504 20:01:40.757231 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 98590us
W20260504 20:01:40.762388 12791 leader_election.cc:336] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.732s (SENT)
I20260504 20:01:40.775343 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.780803 13524 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:40.782981 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.789559 12824 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:40.742909 15396 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:40.802870 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.812168 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.793149 15477 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:01:40.813480 12747 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:40.787941 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:40.778429 15532 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260504 20:01:40.813517 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:40.790997 15530 raft_consensus.cc:697] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:40.801831 15396 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:40.805426 15405 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:40.821733 13497 leader_election.cc:336] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.934s (SENT)
W20260504 20:01:40.833751 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 202612us
W20260504 20:01:40.837074 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:40.837188 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 93608.3us
W20260504 20:01:40.840883 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:40.846774 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 75531.9us
I20260504 20:01:40.851140 12819 raft_consensus.cc:2410] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260504 20:01:40.836234 15513 log.cc:927] Time spent T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.359s user 0.000s sys 0.002s
I20260504 20:01:40.857264 12747 raft_consensus.cc:1240] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260504 20:01:40.832084 15477 raft_consensus.cc:697] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:40.865322 15495 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:40.872426 12791 leader_election.cc:336] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.734s (SENT)
I20260504 20:01:40.874547 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:40.838268 15534 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260504 20:01:40.846482 15025 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:40.861780 12647 catalog_manager.cc:5671] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 5 to 7, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1). New cstate: current_term: 7 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:01:40.869745 12863 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:40.883214 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:40.863581 15515 log.cc:927] Time spent T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.253s user 0.001s sys 0.000s
I20260504 20:01:40.925570 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:40.930045 12819 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:40.935242 12824 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:40.845326 15530 consensus_queue.cc:237] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:40.822172 15483 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:40.944736 12714 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:40.948280 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 116595us
W20260504 20:01:40.936254 15415 consensus_meta.cc:287] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.665s user 0.012s sys 0.003s
I20260504 20:01:40.955403 12716 leader_election.cc:400] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:01:40.960748 12716 leader_election.cc:403] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260504 20:01:40.955977 12647 catalog_manager.cc:5671] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 3 to 4, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 4 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } }
W20260504 20:01:40.957962 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 145366us
W20260504 20:01:40.950737 15380 consensus_meta.cc:287] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.544s user 0.006s sys 0.007s
I20260504 20:01:40.957647 15390 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:40.868731 15435 consensus_queue.cc:237] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:40.873075 15246 consensus_queue.cc:237] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:40.977831 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:40.959849 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:40.980568 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 124093us
I20260504 20:01:40.978382 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:40.972607 15025 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
W20260504 20:01:41.016340 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 45497us
I20260504 20:01:41.019277 12715 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:41.023248 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:41.007680 12647 catalog_manager.cc:5671] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 reported cstate change: term changed from 2 to 3, leader changed from ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
W20260504 20:01:41.007396 15522 log.cc:927] Time spent T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.186s user 0.001s sys 0.000s
I20260504 20:01:41.035308 13532 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260504 20:01:41.052263 12865 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:41.048267 12791 leader_election.cc:336] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.779s (SENT)
I20260504 20:01:41.050195 12748 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:40.972224 15380 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:41.067477 12791 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:41.026160 15550 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:41.047669 15025 raft_consensus.cc:697] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:41.010329 15477 consensus_queue.cc:237] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:41.063295 15488 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.272s
W20260504 20:01:41.334434 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 297990us
I20260504 20:01:41.336153 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:41.292910 15116 consensus_meta.cc:287] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.654s user 0.013s sys 0.002s
W20260504 20:01:41.336752 15533 log.cc:927] Time spent T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.403s user 0.002s sys 0.000s
W20260504 20:01:41.253305 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 458258us
W20260504 20:01:41.290675 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:41.340349 12743 raft_consensus.cc:1240] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
W20260504 20:01:41.219334 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:41.235975 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 224278us
W20260504 20:01:41.347985 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:41.341951 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:41.339280 15116 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:41.354594 13497 leader_election.cc:336] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.663s (SENT)
W20260504 20:01:41.358606 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 339396us
W20260504 20:01:41.252295 15545 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.118s user 0.000s sys 0.006s
W20260504 20:01:41.275415 15544 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
W20260504 20:01:41.256619 15541 log.cc:927] Time spent T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.294s user 0.001s sys 0.001s
W20260504 20:01:41.420104 15364 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.703s user 0.011s sys 0.000s
I20260504 20:01:41.420706 15364 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:41.421151 15025 consensus_queue.cc:237] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:41.423396 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 68872.3us
W20260504 20:01:41.424476 13497 leader_election.cc:336] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.168s (SENT)
I20260504 20:01:41.424757 12865 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:41.128865 15538 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.248s user 0.000s sys 0.002s
W20260504 20:01:41.294745 15009 consensus_meta.cc:287] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.704s user 0.019s sys 0.007s
I20260504 20:01:41.082839 15461 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260504 20:01:41.392181 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 527331us
I20260504 20:01:41.207163 15465 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:41.129153 15543 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260504 20:01:41.095043 15531 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:01:41.026762 15454 consensus_queue.cc:1059] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.336s
I20260504 20:01:41.460660 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:41.469596 13497 leader_election.cc:336] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.840s (SENT)
I20260504 20:01:41.470185 13497 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:41.368542 12749 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.008s sys 0.014s
I20260504 20:01:41.463104 12890 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260504 20:01:41.410902 12746 raft_consensus.cc:1240] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:41.361222 15327 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:41.415509 15546 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:41.401701 15415 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
W20260504 20:01:41.456362 15386 consensus_meta.cc:287] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.004s user 0.009s sys 0.003s
I20260504 20:01:41.367007 15550 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:41.466004 15478 consensus_meta.cc:287] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.137s user 0.017s sys 0.000s
I20260504 20:01:41.130501 15547 consensus_queue.cc:1059] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.418s
W20260504 20:01:41.470889 12749 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s user 0.008s sys 0.015s
I20260504 20:01:41.410379 12892 raft_consensus.cc:3055] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
W20260504 20:01:41.471825 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 445214us
W20260504 20:01:41.472625 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:41.069499 15390 raft_consensus.cc:697] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:41.490401 12894 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 444) took 1016 ms. Trace:
I20260504 20:01:41.470079 12824 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 2.
W20260504 20:01:41.464787 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:41.519615 12894 rpcz_store.cc:276] 0504 20:01:40.457007 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.650094 (+193087us) service_pool.cc:224] Handling call
0504 20:01:40.669754 (+ 19660us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.670022 (+ 268us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:40.670055 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:40.680673 (+ 10618us) log.cc:844] Serialized 26 byte log entry
0504 20:01:40.758516 (+ 77843us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:40.758579 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:40.760323 (+ 1744us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:41.420366 (+660043us) raft_consensus.cc:1685] finished
0504 20:01:41.420388 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:41.473100 (+ 52712us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":15865,"threads_started":1,"wal-append.queue_time_us":209620}
W20260504 20:01:41.446220 15345 consensus_meta.cc:287] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.207s user 0.009s sys 0.008s
W20260504 20:01:41.464058 15555 log.cc:927] Time spent T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
W20260504 20:01:41.467942 15400 consensus_meta.cc:287] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.060s user 0.010s sys 0.006s
W20260504 20:01:41.514565 12895 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.000s sys 0.023s
I20260504 20:01:41.513128 12892 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:41.549512 12895 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s user 0.000s sys 0.023s
I20260504 20:01:41.553231 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 372) took 1293 ms. Trace:
I20260504 20:01:41.554252 12743 rpcz_store.cc:276] 0504 20:01:40.259977 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.260165 (+ 188us) service_pool.cc:224] Handling call
0504 20:01:40.260656 (+ 491us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:41.346959 (+1086303us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:41.552850 (+205891us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b54001ab154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:41.553135 (+ 285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2293856640}
I20260504 20:01:41.470530 15461 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:41.508225 15432 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
W20260504 20:01:41.565265 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 95715.4us
W20260504 20:01:41.562072 12791 leader_election.cc:336] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.838s (SENT)
I20260504 20:01:41.562206 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1807) took 1033 ms. Trace:
I20260504 20:01:41.508201 15523 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:41.523774 15411 consensus_meta.cc:287] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.753s user 0.003s sys 0.009s
I20260504 20:01:41.465126 15373 ts_tablet_manager.cc:1929] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260504 20:01:41.566864 12746 rpcz_store.cc:276] 0504 20:01:40.528769 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.546509 (+ 17740us) service_pool.cc:224] Handling call
0504 20:01:40.602225 (+ 55716us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:41.475916 (+873691us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:41.561853 (+ 85937us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b5400028354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:41.562108 (+ 255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1859180800}
I20260504 20:01:41.508036 15327 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:41.577936 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:41.516743 15524 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:41.579209 15327 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:41.579346 15524 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:41.580538 15478 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:41.581331 15478 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:41.582147 13530 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
W20260504 20:01:41.583223 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 58823.4us
W20260504 20:01:41.584074 12791 leader_election.cc:336] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.844s (SENT)
I20260504 20:01:41.584261 15478 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:41.584547 12791 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:41.589110 15465 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 20:01:41.589332 12791 leader_election.cc:336] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.851s (SENT)
I20260504 20:01:41.589924 12791 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:41.591302 12791 leader_election.cc:336] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.897s (SENT)
I20260504 20:01:41.591755 12791 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:41.592007 15465 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:41.595099 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 655730us
I20260504 20:01:41.598932 15432 raft_consensus.cc:697] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:41.601882 15400 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:41.602319 15432 consensus_queue.cc:237] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:41.603509 12791 leader_election.cc:336] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.987s (SENT)
W20260504 20:01:41.604823 12791 leader_election.cc:336] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.022s (SENT)
W20260504 20:01:41.604998 15396 consensus_meta.cc:287] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.747s user 0.012s sys 0.000s
I20260504 20:01:41.605326 12791 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:41.575145 15380 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:41.605551 15396 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:41.508476 15009 consensus_queue.cc:1059] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260504 20:01:41.564878 15461 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:41.548249 15345 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:41.595281 15411 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:41.636612 12716 connection.cc:254] RPC call timeout handler was delayed by -1.5093292700000802s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.41890000000000005s for a response
W20260504 20:01:41.638278 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:41.623891 15467 consensus_meta.cc:287] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.006s user 0.015s sys 0.000s
W20260504 20:01:41.646694 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 255378us
I20260504 20:01:41.623486 15523 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:41.651897 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:41.656363 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:41.688835 13528 raft_consensus.cc:1240] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:41.635150 15465 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:41.640910 15009 consensus_queue.cc:1243] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer ef6057d9a49e4942b47ea0fcd66ab6f7 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260504 20:01:41.643177 15373 log.cc:1199] T 91ff767ed7c24b97940a459421881055 P 732c3560a688494196d22a1a48f89968: Deleting WAL directory at /tmp/dist-test-taskXcyRki/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777924745039007-11536-0/minicluster-data/ts-2-root/wals/91ff767ed7c24b97940a459421881055
W20260504 20:01:41.697540 12716 leader_election.cc:336] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.933s (SENT)
W20260504 20:01:41.692898 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 47344.3us
I20260504 20:01:41.705142 15411 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:41.707258 15535 raft_consensus.cc:3055] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:41.707813 15535 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:41.708809 15535 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:41.710605 15396 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:41.711387 15364 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:41.719492 12716 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:41.724381 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:41.725119 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 146601us
I20260504 20:01:41.725335 12743 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:41.726585 12746 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260504 20:01:41.728127 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:41.736510 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:41.740422 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:41.744551 13497 leader_election.cc:336] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.525s (SENT)
W20260504 20:01:41.785292 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 180510us
I20260504 20:01:41.786376 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:41.657721 15467 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:41.745337 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:41.744378 12863 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:41.649286 15390 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:41.692104 15523 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:41.811682 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:41.903784 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:41.746243 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:42.011102 13528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 415) took 1509 ms. Trace:
I20260504 20:01:42.013579 13528 rpcz_store.cc:276] 0504 20:01:40.501988 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.502161 (+ 173us) service_pool.cc:224] Handling call
0504 20:01:40.520377 (+ 18216us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:41.689478 (+1169101us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:42.005180 (+315702us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540049ded4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:42.011026 (+ 5846us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2686628608}
I20260504 20:01:42.024281 12819 raft_consensus.cc:2410] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:01:42.026182 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:41.993733 15524 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:42.025197 15537 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:42.033254 15537 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:42.033676 15537 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:41.986016 15583 log.cc:927] Time spent T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20260504 20:01:42.025081 12890 raft_consensus.cc:2468] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 2.
I20260504 20:01:42.028172 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 446) took 1359 ms. Trace:
W20260504 20:01:42.038463 15570 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.336s user 0.001s sys 0.000s
W20260504 20:01:42.038843 15423 consensus_meta.cc:287] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.407s user 0.017s sys 0.000s
I20260504 20:01:42.004078 12892 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:42.038658 12896 rpcz_store.cc:276] 0504 20:01:40.650356 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.661650 (+ 11294us) service_pool.cc:224] Handling call
0504 20:01:40.679071 (+ 17421us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.692400 (+ 13329us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:40.692445 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:40.702946 (+ 10501us) log.cc:844] Serialized 26 byte log entry
0504 20:01:41.048477 (+345531us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:41.048553 (+ 76us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:41.048634 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:42.009494 (+960860us) raft_consensus.cc:1685] finished
0504 20:01:42.009524 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:42.009713 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":275870,"threads_started":1,"wal-append.queue_time_us":554427}
I20260504 20:01:42.040285 15373 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37306 (request call id 338) took 8962 ms. Trace:
I20260504 20:01:42.040535 12892 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:42.042302 13527 raft_consensus.cc:2410] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260504 20:01:42.043867 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 616) took 1344 ms. Trace:
W20260504 20:01:42.044984 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 200063us
W20260504 20:01:42.028738 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 661973us
W20260504 20:01:41.917677 12791 leader_election.cc:336] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.060s (SENT)
I20260504 20:01:41.904268 12817 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8. Ops: []
W20260504 20:01:41.879688 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.044296 13527 rpcz_store.cc:276] 0504 20:01:40.699693 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.716475 (+ 16782us) service_pool.cc:224] Handling call
0504 20:01:42.043608 (+1327133us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540043d354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:42.043792 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2797885312}
I20260504 20:01:42.008103 15483 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:41.744925 15009 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:41.936750 15467 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:41.736124 15532 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:41.989984 15386 raft_consensus.cc:3055] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
W20260504 20:01:42.031857 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:42.065965 13530 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 609) took 1832 ms (client timeout 2055 ms). Trace:
W20260504 20:01:42.007902 15565 log.cc:927] Time spent T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.358s user 0.001s sys 0.000s
I20260504 20:01:42.040788 15373 rpcz_store.cc:276] 0504 20:01:33.115012 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:33.170875 (+ 55863us) service_pool.cc:224] Handling call
0504 20:01:33.693212 (+522337us) ts_tablet_manager.cc:1151] Deleting tablet 91ff767ed7c24b97940a459421881055
0504 20:01:33.796214 (+103002us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 20:01:39.486869 (+5690655us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b540035c0d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b6742 00007f01603ac7bc 00007f01619d2cca 00007f0161a03990 00007f01619e9efd 00007f01619e9ea9 00007f01619e9e39 00007f01619e9e01 00007f01619e902d 0000000000379254
0504 20:01:40.467265 (+980396us) tablet_metadata.cc:665] Metadata flushed
0504 20:01:41.464951 (+997686us) tablet_metadata.cc:665] Metadata flushed
0504 20:01:42.040172 (+575221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3270378,"spinlock_wait_cycles":4650351488,"tablet-delete.queue_time_us":368465,"thread_start_us":179349,"threads_started":1}
W20260504 20:01:42.008806 15568 log.cc:927] Time spent T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.321s user 0.001s sys 0.000s
W20260504 20:01:41.942421 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 251379us
W20260504 20:01:42.069905 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:42.111537 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:42.010718 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 266476us
W20260504 20:01:41.990908 15574 log.cc:927] Time spent T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260504 20:01:41.975262 15550 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:41.902673 15543 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:42.086268 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 341747us
I20260504 20:01:42.089064 15423 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:42.089720 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 43889us
W20260504 20:01:42.091095 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 173471us
I20260504 20:01:42.094089 15560 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:41.937896 15345 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:42.006947 15561 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.373s user 0.001s sys 0.000s
I20260504 20:01:42.193739 15560 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.108142 12743 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:41.844367 15415 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.240s
W20260504 20:01:42.082041 15569 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
W20260504 20:01:42.196903 15587 log.cc:927] Time spent T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260504 20:01:42.098088 15524 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:42.115700 12865 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:42.096002 15386 raft_consensus.cc:740] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:42.119746 12817 raft_consensus.cc:1275] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260504 20:01:42.119829 12716 leader_election.cc:336] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.978s (SENT)
I20260504 20:01:42.203742 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:42.128284 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:42.081486 15009 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.178532 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:42.204983 12818 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:42.177098 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:42.186401 15179 consensus_meta.cc:287] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.508s user 0.022s sys 0.000s
I20260504 20:01:42.129419 15495 consensus_queue.cc:1059] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260504 20:01:42.188934 15439 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:01:42.190016 15380 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:42.190852 13499 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:42.192030 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1816) took 1252 ms. Trace:
I20260504 20:01:42.196014 12750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1810) took 1527 ms. Trace:
W20260504 20:01:42.100317 13530 rpcz_store.cc:269] 0504 20:01:40.233277 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.254165 (+ 20888us) service_pool.cc:224] Handling call
0504 20:01:42.065686 (+1811521us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54006efd54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:42.065874 (+ 188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3051523456}
W20260504 20:01:42.212980 12791 leader_election.cc:336] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.065s (SENT)
I20260504 20:01:42.220346 12791 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:42.214226 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:42.201342 15565 log.cc:927] Time spent T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.086s user 0.000s sys 0.006s
I20260504 20:01:42.215632 12747 rpcz_store.cc:276] 0504 20:01:40.939760 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.942209 (+ 2449us) service_pool.cc:224] Handling call
0504 20:01:40.959056 (+ 16847us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:41.079655 (+120599us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:41.079700 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:41.136591 (+ 56891us) log.cc:844] Serialized 26 byte log entry
0504 20:01:41.467623 (+331032us) spinlock_profiling.cc:243] Waited 8.24 ms on lock 0x7b4800300258. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f01618e61a1 00007f01618dd237 00007f01618dd0ca 00007f01619d492f 00007f01619d45f7 00007f01619e8c1c 00007f01619e8bc8 00007f01619e8b48 00007f01619e8b03 00007f01619e7dac 00007f0160910b36
0504 20:01:41.563832 (+ 96209us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:41.563879 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:41.563962 (+ 83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:42.191716 (+627754us) raft_consensus.cc:1685] finished
0504 20:01:42.191740 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:42.191961 (+ 221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78768,"raft.queue_time_us":107354,"raft.run_cpu_time_us":3206,"raft.run_wall_time_us":374532,"spinlock_wait_cycles":18967808,"thread_start_us":222621,"threads_started":1,"wal-append.queue_time_us":454295}
I20260504 20:01:42.217355 13499 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:42.205835 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:42.233851 12635 catalog_manager.cc:5002] TS 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): tablet 91ff767ed7c24b97940a459421881055 (table test-workload [id=8248412214354c89a76d2a364e6262e0]) successfully deleted
I20260504 20:01:42.235119 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:42.239612 12890 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:42.235925 15536 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:42.238071 15566 raft_consensus.cc:3055] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:01:42.241308 12821 raft_consensus.cc:1275] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:01:42.223672 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:42.223006 15380 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:42.242887 12818 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:42.212596 15179 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.215727 12750 rpcz_store.cc:276] 0504 20:01:40.668377 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.670845 (+ 2468us) service_pool.cc:224] Handling call
0504 20:01:40.691807 (+ 20962us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.779542 (+ 87735us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:40.779589 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:40.822985 (+ 43396us) log.cc:844] Serialized 26 byte log entry
0504 20:01:40.868247 (+ 45262us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b1800406000. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d471 00007f015c969def
0504 20:01:41.228856 (+360609us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:41.228910 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:41.228998 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:42.195714 (+966716us) raft_consensus.cc:1685] finished
0504 20:01:42.195735 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:42.195946 (+ 211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20343,"raft.queue_time_us":80057,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":27613,"spinlock_wait_cycles":5627648,"thread_start_us":422604,"threads_started":2,"wal-append.queue_time_us":725681}
I20260504 20:01:42.190732 15523 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:42.206319 15540 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:42.282464 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 69005.6us
W20260504 20:01:42.375499 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.201853 15423 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:42.283186 13497 leader_election.cc:336] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.280s (SENT)
W20260504 20:01:42.285959 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 166184us
W20260504 20:01:42.286834 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 172261us
I20260504 20:01:42.287920 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:42.196255 15560 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:42.287490 15566 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:01:42.320493 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 97504.9us
W20260504 20:01:42.321890 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 111761us
I20260504 20:01:42.242115 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 448) took 1559 ms. Trace:
I20260504 20:01:42.329695 12892 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260504 20:01:42.382936 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:42.383607 12892 pending_rounds.cc:85] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 5
I20260504 20:01:42.383950 12892 pending_rounds.cc:107] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260504 20:01:42.318677 15386 consensus_queue.cc:260] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.384312 12892 raft_consensus.cc:2889] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260504 20:01:42.384888 12716 leader_election.cc:336] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.035s (SENT)
I20260504 20:01:42.334316 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1812) took 1588 ms. Trace:
W20260504 20:01:42.385741 15563 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260504 20:01:42.386706 15396 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:42.387233 15396 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:42.387648 15423 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:42.385528 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 450) took 1021 ms. Trace:
I20260504 20:01:42.388864 12892 rpcz_store.cc:276] 0504 20:01:41.363723 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:41.364107 (+ 384us) service_pool.cc:224] Handling call
0504 20:01:41.403493 (+ 39386us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:42.003483 (+599990us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54002943a0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c92c44f 00007f015c9a5340 00007f015c9a6bbe 00007f015c9b0f29 00007f015c9b17f6 00007f015c9ae16f 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2
0504 20:01:42.385042 (+381559us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:42.385428 (+ 386us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":329284480}
I20260504 20:01:42.335050 15590 raft_consensus.cc:2804] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:42.390518 15590 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:42.390762 15009 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:42.391001 15590 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:42.392889 15566 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:42.394754 12890 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:01:42.395241 13531 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260504 20:01:42.383131 12895 rpcz_store.cc:276] 0504 20:01:40.682968 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.683250 (+ 282us) service_pool.cc:224] Handling call
0504 20:01:40.710652 (+ 27402us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.936446 (+225794us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:40.936492 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:40.943772 (+ 7280us) log.cc:844] Serialized 26 byte log entry
0504 20:01:41.550065 (+606293us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:41.550125 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:41.550213 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:42.241881 (+691668us) raft_consensus.cc:1685] finished
0504 20:01:42.241902 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:42.242049 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":93088,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":18260,"thread_start_us":785215,"threads_started":2,"wal-append.queue_time_us":621708}
W20260504 20:01:42.336097 15586 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260504 20:01:42.336539 15585 log.cc:927] Time spent T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
W20260504 20:01:42.339160 15592 log.cc:927] Time spent T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260504 20:01:42.340118 15581 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260504 20:01:42.340449 15535 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:42.340989 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 134741us
I20260504 20:01:42.341536 12747 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:01:42.407534 12747 raft_consensus.cc:606] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:42.407938 12747 raft_consensus.cc:614] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:42.342058 15571 raft_consensus.cc:2804] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:42.342532 12863 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:42.347072 15380 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:42.347532 15488 consensus_queue.cc:1059] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260504 20:01:42.350270 15246 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:42.353075 15461 consensus_meta.cc:287] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.662s user 0.011s sys 0.000s
I20260504 20:01:42.448027 15461 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.450448 15461 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:42.212819 15543 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.452663 15543 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:42.285761 15536 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:42.453758 15536 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:42.454511 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 171364us
W20260504 20:01:42.455296 13497 leader_election.cc:336] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.991s (SENT)
I20260504 20:01:42.455747 13497 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:42.456907 15423 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:42.457206 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.301154 15465 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:42.371520 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:42.385979 12745 rpcz_store.cc:276] 0504 20:01:40.745981 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.746153 (+ 172us) service_pool.cc:224] Handling call
0504 20:01:40.782609 (+ 36456us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:40.850770 (+ 68161us) raft_consensus.cc:1534] Early marking committed up to index 4
0504 20:01:40.892318 (+ 41548us) log.cc:844] Serialized 14 byte log entry
0504 20:01:41.253273 (+360955us) spinlock_profiling.cc:243] Waited 5.76 ms on lock 0x7b180037dbc0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d5da 00007f015c9b8e0a
0504 20:01:41.447520 (+194247us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:41.508596 (+ 61076us) log.cc:844] Serialized 26 byte log entry
0504 20:01:41.508783 (+ 187us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:41.508820 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:41.508902 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:42.333888 (+824986us) raft_consensus.cc:1685] finished
0504 20:01:42.334000 (+ 112us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:42.334190 (+ 190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32423,"prepare.queue_time_us":83806,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":2882,"raft.queue_time_us":117621,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":2819,"spinlock_wait_cycles":13257600,"thread_start_us":220213,"threads_started":2,"wal-append.queue_time_us":642282}
W20260504 20:01:42.334702 15589 log.cc:927] Time spent T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260504 20:01:42.385520 12716 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:42.404320 13499 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:42.401830 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1813) took 1597 ms. Trace:
W20260504 20:01:42.419930 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 77505.8us
I20260504 20:01:42.401052 15537 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.421622 15025 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:01:42.380015 15540 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:42.418723 15524 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260504 20:01:42.383206 15179 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:42.387651 15396 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:42.423309 15009 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:42.425302 15386 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:42.423364 15594 raft_consensus.cc:3055] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Stepping down as leader of term 5
W20260504 20:01:42.434258 12791 leader_election.cc:336] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.090s (SENT)
I20260504 20:01:42.435581 15571 raft_consensus.cc:697] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:42.438335 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:42.445726 12896 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:42.446368 15560 consensus_queue.cc:1048] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:42.468456 15566 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:42.468389 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:42.519349 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:42.519843 12745 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:42.469848 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:42.470950 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:42.521319 12744 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 9.
I20260504 20:01:42.474865 13531 consensus_queue.cc:237] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:42.482542 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:42.481298 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 77076.5us
I20260504 20:01:42.481227 12749 rpcz_store.cc:276] 0504 20:01:40.804448 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:40.804620 (+ 172us) service_pool.cc:224] Handling call
0504 20:01:40.812774 (+ 8154us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:41.523224 (+710450us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:41.523256 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:41.570109 (+ 46853us) log.cc:844] Serialized 26 byte log entry
0504 20:01:42.010054 (+439945us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:42.010095 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:42.010173 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:42.401563 (+391390us) raft_consensus.cc:1685] finished
0504 20:01:42.401584 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:42.401735 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8905,"raft.queue_time_us":46041,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":825377,"threads_started":2,"wal-append.queue_time_us":574386}
I20260504 20:01:42.497687 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:42.503859 12821 raft_consensus.cc:1275] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260504 20:01:42.469043 15573 log.cc:927] Time spent T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.530s user 0.004s sys 0.002s
I20260504 20:01:42.510959 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:42.526813 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:42.515727 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 130908us
W20260504 20:01:42.458693 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.485538 15246 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:42.468941 15400 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:42.493597 15423 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:42.492715 15535 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:42.520066 12825 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:42.536761 12750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "91ff767ed7c24b97940a459421881055"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:01:42.532207 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:42.533654 13497 leader_election.cc:336] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.562s (SENT)
I20260504 20:01:42.542133 13497 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:42.542872 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.517704 15483 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:42.544097 13499 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:42.514567 15594 raft_consensus.cc:740] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:01:42.549834 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 111583us
I20260504 20:01:42.553058 12819 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:42.556571 13499 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:42.547498 12716 leader_election.cc:336] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.125s (SENT)
I20260504 20:01:42.531749 15009 raft_consensus.cc:697] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:42.564237 12716 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:42.531569 15543 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:42.541088 15563 raft_consensus.cc:2804] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
W20260504 20:01:42.566514 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 132384us
W20260504 20:01:42.573551 12791 leader_election.cc:336] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.127s (SENT)
W20260504 20:01:42.574442 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:42.576917 13497 leader_election.cc:336] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.077s (SENT)
I20260504 20:01:42.577597 13497 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:42.582269 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 199913us
I20260504 20:01:42.559643 15423 raft_consensus.cc:697] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:42.595122 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:42.579456 15563 raft_consensus.cc:697] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:42.605968 15580 consensus_queue.cc:1048] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.099s
I20260504 20:01:42.607987 15593 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:01:42.546646 15495 consensus_queue.cc:1048] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260504 20:01:42.579154 15415 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:42.572249 15439 raft_consensus.cc:2804] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:42.545145 15400 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:42.573351 15539 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:42.558569 15483 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.631111 12819 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:42.639938 15411 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:42.646389 15423 consensus_queue.cc:237] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:42.647773 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 108122us
W20260504 20:01:42.649132 12716 leader_election.cc:336] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.224s (SENT)
W20260504 20:01:42.650019 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 80784.9us
W20260504 20:01:42.650161 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 75682.5us
I20260504 20:01:42.625895 15593 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:42.577383 15543 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.550944 15594 consensus_queue.cc:260] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:42.628821 15439 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:42.650123 15372 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260504 20:01:42.671907 12791 leader_election.cc:336] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.127s (SENT)
I20260504 20:01:42.630030 15571 consensus_queue.cc:237] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:42.681145 15594 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:42.683382 15540 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:42.654353 15179 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:42.684284 15179 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:42.622368 15590 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:42.687073 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:42.687302 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.658950 15593 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:01:42.695490 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:42.672211 15439 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:42.699750 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:42.671295 15537 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:42.703061 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 53969us
W20260504 20:01:42.707660 12716 connection.cc:254] RPC call timeout handler was delayed by -1.0143608360000371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18730000000000002s for a response
I20260504 20:01:42.713892 12750 raft_consensus.cc:606] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:42.714440 12750 raft_consensus.cc:614] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:42.713267 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:42.715884 12820 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 4
W20260504 20:01:42.716435 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:42.720685 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 103366us
W20260504 20:01:42.724376 12716 leader_election.cc:336] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.350s (SENT)
I20260504 20:01:42.726733 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:42.730545 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 63577.4us
I20260504 20:01:42.733940 12716 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:42.671484 15009 consensus_queue.cc:237] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.712483 15540 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:42.747354 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 75847.1us
I20260504 20:01:42.748576 12865 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:42.711877 15539 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:42.763831 15540 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.746675 15537 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:42.785722 12791 leader_election.cc:336] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.154s (SENT)
I20260504 20:01:42.786370 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:42.750979 15396 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:42.718135 15563 consensus_queue.cc:237] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:42.844843 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.751876 15465 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:42.819080 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:42.943817 15405 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:42.944226 15537 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:42.946060 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 172895us
W20260504 20:01:42.947119 12791 leader_election.cc:336] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.186s (SENT)
I20260504 20:01:42.947902 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:42.906721 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.847227 15179 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:42.935154 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 129350us
W20260504 20:01:42.816609 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 92244.3us
W20260504 20:01:42.958684 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 381787us
W20260504 20:01:42.948110 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 199661us
W20260504 20:01:42.961205 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:42.951473 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:42.963583 15488 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:42.965976 12791 leader_election.cc:336] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.221s (SENT)
I20260504 20:01:42.968537 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:42.972254 15467 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:42.976948 12791 connection.cc:254] RPC call timeout handler was delayed by -5.0484664550000389s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2821s for a response
W20260504 20:01:42.981163 12791 connection.cc:254] RPC call timeout handler was delayed by -3.8130333979999023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
I20260504 20:01:42.771097 15483 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:42.767201 15535 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260504 20:01:42.995645 12791 connection.cc:254] RPC call timeout handler was delayed by -2.6043333979999943s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30190000000000006s for a response
W20260504 20:01:42.997354 12716 leader_election.cc:336] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.450s (SENT)
I20260504 20:01:43.005663 12716 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:42.999663 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 279054us
I20260504 20:01:43.004065 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:42.942492 15540 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:42.949141 15560 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:43.004374 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:43.023052 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.009096 12863 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:43.025071 15601 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.322s user 0.001s sys 0.000s
W20260504 20:01:43.032763 15608 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260504 20:01:42.779062 15539 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:43.044781 12863 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:01:43.050601 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:43.051174 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.019717 15590 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:43.041767 15536 consensus_meta.cc:287] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.571s user 0.005s sys 0.008s
W20260504 20:01:43.067234 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.024585 15534 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:43.052220 12865 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:43.041932 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:43.040180 15543 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:43.051303 15179 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260504 20:01:43.097430 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 45914.2us
I20260504 20:01:43.106454 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:43.112241 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 152586us
W20260504 20:01:43.113453 12716 leader_election.cc:336] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 3.286s (SENT)
I20260504 20:01:43.113953 12716 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:43.116518 15606 log.cc:927] Time spent T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20260504 20:01:43.116706 15400 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:43.118973 15602 log.cc:927] Time spent T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260504 20:01:43.093407 15536 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:43.121130 15411 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:43.096915 15622 raft_consensus.cc:2804] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:43.123199 15483 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260504 20:01:43.124351 15386 consensus_meta.cc:287] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.603s user 0.012s sys 0.000s
I20260504 20:01:43.126053 15396 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:43.126935 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.131206 12895 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:43.121522 15536 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:43.150699 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:43.153373 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:43.153203 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:43.106626 15179 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:43.158032 15534 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:43.158444 15179 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:43.162983 12863 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:43.166188 15432 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:43.166571 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 140519us
I20260504 20:01:43.169356 12865 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:43.163038 15501 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:43.171779 15604 log.cc:927] Time spent T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.501s user 0.003s sys 0.000s
W20260504 20:01:43.256811 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 152000us
I20260504 20:01:43.055676 15560 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:43.128643 15537 raft_consensus.cc:2804] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:43.144212 15622 raft_consensus.cc:697] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:43.283975 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 101715us
W20260504 20:01:43.287120 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 173729us
W20260504 20:01:43.298264 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.291416 15386 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:43.338152 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 168890us
I20260504 20:01:43.228775 15501 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:43.288842 15537 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:43.302486 15543 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:43.356274 12750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:01:43.357556 12750 raft_consensus.cc:606] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:43.358129 12750 raft_consensus.cc:614] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260504 20:01:43.342525 15618 log.cc:927] Time spent T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260504 20:01:43.339144 15542 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:43.370496 12820 consensus_meta.cc:287] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.649s user 0.011s sys 0.003s
I20260504 20:01:43.349145 15537 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:43.371042 12820 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
W20260504 20:01:43.375168 15566 consensus_meta.cc:287] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.831s user 0.015s sys 0.000s
I20260504 20:01:43.390916 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260504 20:01:43.392541 12865 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:43.368688 15605 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:43.396273 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.393687 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260504 20:01:43.384507 15593 consensus_meta.cc:287] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.675s user 0.010s sys 0.004s
I20260504 20:01:43.374311 15539 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:43.356285 15543 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:43.400593 13524 raft_consensus.cc:1240] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:43.407565 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:43.410254 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 110685us
I20260504 20:01:43.379415 15542 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:43.431576 15625 log.cc:927] Time spent T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
W20260504 20:01:43.433501 15620 log.cc:927] Time spent T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20260504 20:01:43.433640 15635 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:43.437306 12895 raft_consensus.cc:2468] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:43.441491 12743 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:43.446791 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.447788 12819 raft_consensus.cc:2410] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260504 20:01:43.401613 15605 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:43.451927 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.459836 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.460808 12818 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:01:43.469318 12819 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
W20260504 20:01:43.477345 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.439859 15540 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:43.478942 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.408449 15593 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:43.427618 15534 consensus_queue.cc:1059] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260504 20:01:43.480072 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:43.491025 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.528074 12820 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:43.398566 15622 consensus_queue.cc:237] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:43.531404 12714 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:43.446070 15635 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:43.538714 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:43.538854 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.510836 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:43.534904 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.542846 15636 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:43.543581 15636 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:43.544067 15636 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:43.546528 12821 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260504 20:01:43.547314 15593 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:43.522897 15633 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
I20260504 20:01:43.551757 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:43.479604 15638 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
W20260504 20:01:43.569988 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.532321 15501 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:43.503587 15540 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:43.573295 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.680028 13499 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:43.614243 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 208302us
I20260504 20:01:43.677784 13532 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:43.674549 13528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 347) took 1103 ms. Trace:
I20260504 20:01:43.674618 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:43.584230 12820 raft_consensus.cc:1240] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:43.682135 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:43.611001 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 230138us
I20260504 20:01:43.682895 13533 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
W20260504 20:01:43.589574 12716 leader_election.cc:336] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.585s (SENT)
I20260504 20:01:43.588244 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.681579 13528 rpcz_store.cc:276] 0504 20:01:42.574092 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:42.578832 (+ 4740us) service_pool.cc:224] Handling call
0504 20:01:42.610267 (+ 31435us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:42.711554 (+101287us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:42.712378 (+ 824us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:43.434487 (+722109us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b2c00112268. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015b2a8c81 00007f015b2b85e2 00007f015c9e9de1 00007f015c9ae81a 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22
0504 20:01:43.435037 (+ 550us) log.cc:844] Serialized 26 byte log entry
0504 20:01:43.532190 (+ 97153us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:43.532259 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:43.532343 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:43.674156 (+141813us) raft_consensus.cc:1685] finished
0504 20:01:43.674193 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:43.674472 (+ 279us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7060,"raft.queue_time_us":33468,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":5169,"spinlock_wait_cycles":14422400,"thread_start_us":188858,"threads_started":2,"wal-append.queue_time_us":235154}
W20260504 20:01:43.709952 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 331661us
I20260504 20:01:43.550311 15560 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:43.711635 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 51792us
W20260504 20:01:43.741240 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 151725us
W20260504 20:01:43.747710 12716 leader_election.cc:336] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.915s (SENT)
I20260504 20:01:43.742149 12865 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:43.666803 15598 log.cc:927] Time spent T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.369s user 0.003s sys 0.000s
W20260504 20:01:43.664300 15612 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.994s user 0.006s sys 0.001s
I20260504 20:01:43.745117 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:43.757007 15439 consensus_meta.cc:287] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.044s user 0.017s sys 0.006s
W20260504 20:01:43.669843 15632 log.cc:927] Time spent T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.242s user 0.000s sys 0.001s
W20260504 20:01:43.751155 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.751287 12863 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:43.753634 15560 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:43.711048 15643 raft_consensus.cc:2804] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:43.755647 15635 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:43.757114 12818 raft_consensus.cc:2368] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5.
I20260504 20:01:43.757489 15439 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:43.758318 15542 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:43.653680 15386 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:43.764114 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:43.766103 12819 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:43.589089 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:43.754668 15605 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:43.609529 15543 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:43.769851 12865 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:43.779954 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.781630 12818 raft_consensus.cc:1240] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:43.784469 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:43.784682 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:43.794253 13533 raft_consensus.cc:2410] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260504 20:01:43.744601 15501 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260504 20:01:43.796161 13531 raft_consensus.cc:2368] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
W20260504 20:01:43.791240 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 43567us
I20260504 20:01:43.791324 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:43.796868 12716 leader_election.cc:336] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.015s (SENT)
I20260504 20:01:43.797276 12716 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:43.776098 15396 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:43.801602 12865 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:43.803050 12865 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:43.780961 12715 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:43.792695 15539 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:43.824412 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:43.825299 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 267161us
W20260504 20:01:43.826681 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 45830.4us
I20260504 20:01:43.791846 15643 raft_consensus.cc:697] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:43.799748 15621 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:43.775761 15560 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:43.846089 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.844673 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:43.845116 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.855443 12825 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
W20260504 20:01:43.836669 12716 leader_election.cc:336] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.168s (SENT)
I20260504 20:01:43.854539 12743 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:01:43.856240 12716 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:43.849727 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:43.809098 15364 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:43.826347 15534 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:43.827035 15536 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:43.846959 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:43.838467 15631 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.399s user 0.003s sys 0.001s
I20260504 20:01:43.848166 15621 raft_consensus.cc:697] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:43.828415 15386 raft_consensus.cc:697] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:43.848769 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.841529 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.842684 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:43.854506 12824 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:01:43.881474 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:43.882516 12895 raft_consensus.cc:2410] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260504 20:01:43.855326 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.795493 13527 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:43.890422 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.891299 12816 raft_consensus.cc:2410] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260504 20:01:43.857203 12892 raft_consensus.cc:2368] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:43.841149 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.831758 12818 raft_consensus.cc:2393] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:43.823081 15025 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:43.812811 15543 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 20:01:43.900566 15560 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:43.872463 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.873447 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:43.909569 12817 raft_consensus.cc:2410] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260504 20:01:43.910526 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.911636 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.859655 15364 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:43.912453 15364 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:43.912777 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 462) took 1188 ms. Trace:
I20260504 20:01:43.914175 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:43.913213 12898 rpcz_store.cc:276] 0504 20:01:42.724153 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:42.724538 (+ 385us) service_pool.cc:224] Handling call
0504 20:01:42.739258 (+ 14720us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:42.770487 (+ 31229us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:42.770524 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:42.771084 (+ 560us) log.cc:844] Serialized 26 byte log entry
0504 20:01:43.335286 (+564202us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:43.335356 (+ 70us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:43.335467 (+ 111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:43.744386 (+408919us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:43.912478 (+168092us) raft_consensus.cc:1685] finished
0504 20:01:43.912505 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:43.912706 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30685,"raft.queue_time_us":203910,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":151,"spinlock_wait_cycles":5145344,"thread_start_us":407342,"threads_started":1,"wal-append.queue_time_us":576304}
I20260504 20:01:43.914819 15540 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:43.915926 15439 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:43.880694 15649 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:43.873903 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:43.920228 15179 consensus_meta.cc:287] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.761s user 0.013s sys 0.000s
I20260504 20:01:43.871065 15536 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:43.921160 15536 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:43.922427 15386 consensus_queue.cc:237] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:43.835956 15535 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:43.878461 15534 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:43.894001 12743 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 10.
I20260504 20:01:43.894888 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.897974 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.898910 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.899027 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:43.816776 15563 consensus_queue.cc:1048] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.053s
I20260504 20:01:43.900019 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:43.939637 12824 raft_consensus.cc:2393] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:43.839656 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:43.926736 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.927973 12823 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
W20260504 20:01:43.931967 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 95321.2us
I20260504 20:01:43.934221 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.939962 12818 raft_consensus.cc:2376] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5. Re-sending same reply.
I20260504 20:01:43.953962 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:43.954728 12894 raft_consensus.cc:2393] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:01:43.958137 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:43.938647 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 113167us
I20260504 20:01:43.961416 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.963596 13531 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 623) took 1252 ms. Trace:
I20260504 20:01:43.964030 13531 rpcz_store.cc:276] 0504 20:01:42.711069 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:42.711333 (+ 264us) service_pool.cc:224] Handling call
0504 20:01:43.963364 (+1252031us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54006efd54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:43.963527 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2457440128}
I20260504 20:01:43.966167 12825 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260504 20:01:43.966697 15624 log.cc:927] Time spent T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.681s user 0.004s sys 0.000s
I20260504 20:01:43.967075 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.967896 12893 raft_consensus.cc:2368] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:43.968616 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1931) took 1273 ms. Trace:
I20260504 20:01:43.968786 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.969568 12890 raft_consensus.cc:2393] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 8: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:43.970276 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.970662 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.971387 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:43.972159 12891 raft_consensus.cc:2410] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260504 20:01:43.972916 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.973543 12816 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
W20260504 20:01:43.975746 15537 consensus_meta.cc:287] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.588s user 0.018s sys 0.031s
I20260504 20:01:43.941365 12817 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260504 20:01:43.977496 12817 pending_rounds.cc:85] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 7
I20260504 20:01:43.977746 12817 pending_rounds.cc:107] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260504 20:01:43.953672 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:43.977985 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.978005 12817 raft_consensus.cc:2889] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:43.983285 13499 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:43.990258 12715 leader_election.cc:400] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:43.990789 12715 leader_election.cc:403] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:43.991189 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:43.992182 12899 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:43.969024 12822 rpcz_store.cc:276] 0504 20:01:42.695535 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:42.696822 (+ 1287us) service_pool.cc:224] Handling call
0504 20:01:42.702644 (+ 5822us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:42.731813 (+ 29169us) raft_consensus.cc:1534] Early marking committed up to index 4
0504 20:01:42.765431 (+ 33618us) log.cc:844] Serialized 14 byte log entry
0504 20:01:42.994989 (+229558us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:43.015237 (+ 20248us) log.cc:844] Serialized 26 byte log entry
0504 20:01:43.015472 (+ 235us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:43.015521 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:43.015619 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:43.966056 (+950437us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:43.968299 (+ 2243us) raft_consensus.cc:1685] finished
0504 20:01:43.968407 (+ 108us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:43.968553 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4232,"prepare.queue_time_us":39512,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":57,"raft.queue_time_us":11598,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":31295,"spinlock_wait_cycles":7433984,"thread_start_us":149202,"threads_started":1,"wal-append.queue_time_us":306074}
W20260504 20:01:43.997632 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:43.997725 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.920683 15179 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:43.932660 15535 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:44.002389 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.003252 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:44.007419 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.013310 12894 raft_consensus.cc:2410] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260504 20:01:43.977973 15563 consensus_queue.cc:1243] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer 82787d8e64d24ca885ca517aa8ed1be2 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260504 20:01:44.015345 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:44.014278 13497 leader_election.cc:336] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.795s (SENT)
I20260504 20:01:44.021226 13497 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:43.990310 15539 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:44.019632 12899 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:01:44.023607 12893 raft_consensus.cc:2393] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:44.025112 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.001549 15543 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:44.009482 15535 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:44.027974 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:44.031140 15396 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:44.000692 15642 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Leader election won for term 10
I20260504 20:01:44.019251 15650 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:44.037200 12821 raft_consensus.cc:2376] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7. Re-sending same reply.
I20260504 20:01:44.038818 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.039661 12894 raft_consensus.cc:2368] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8.
I20260504 20:01:44.041489 12823 raft_consensus.cc:1275] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:44.041564 12892 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:44.043645 15650 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:44.044090 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.044711 12893 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:44.045627 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:44.046031 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.046450 12821 raft_consensus.cc:2393] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:44.047277 15566 raft_consensus.cc:3055] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:44.047722 15566 raft_consensus.cc:740] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:44.048478 15566 consensus_queue.cc:260] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:44.051723 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.052146 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:44.052594 12821 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:01:44.038267 15648 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:44.053884 15650 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:44.053412 15648 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:44.056450 15648 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:44.049593 15566 raft_consensus.cc:3060] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:44.041157 15613 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.018s
I20260504 20:01:44.060694 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:44.065153 15648 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:01:44.065757 15648 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:44.023038 15638 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:44.018420 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 427) took 1058 ms. Trace:
I20260504 20:01:44.069067 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:43.976238 15537 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:44.074357 12822 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:44.066169 15648 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:44.005138 15643 consensus_queue.cc:237] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:44.081131 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.079979 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.091182 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:44.113423 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.106551 12816 raft_consensus.cc:2376] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6. Re-sending same reply.
I20260504 20:01:44.142902 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.152595 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:44.126355 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.133244 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:44.154577 12823 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:44.158741 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.160650 12896 raft_consensus.cc:1240] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:44.067432 15642 raft_consensus.cc:697] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:44.165830 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:44.088929 15534 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:44.095423 15179 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:44.140179 15543 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:44.186817 12821 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260504 20:01:44.185424 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.188294 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.087803 15364 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:44.198616 12816 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260504 20:01:44.201745 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:44.202065 13497 leader_election.cc:336] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.668s (SENT)
W20260504 20:01:44.208293 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.277654 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.278175 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.279055 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.279876 12894 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 8: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:44.280912 12890 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:44.194463 15380 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:44.080312 15654 consensus_queue.cc:1048] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.140s
I20260504 20:01:44.219791 15635 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:44.299990 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:44.306167 12743 raft_consensus.cc:2468] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:44.309566 12823 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:01:44.322716 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.323073 15536 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:44.324158 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.324760 12894 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:44.325320 15536 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:44.326114 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.326764 12895 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:44.068486 15621 consensus_queue.cc:237] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:44.282522 15534 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:44.325577 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.338379 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:44.339301 12750 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:44.345373 12818 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:01:44.348310 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.351547 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:44.353070 12816 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:01:44.354964 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.360225 12741 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: []
I20260504 20:01:44.341653 15534 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:44.363458 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:44.363833 12749 raft_consensus.cc:1275] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:01:44.364028 15534 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:44.364665 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:44.364856 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:44.364975 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:44.366115 12748 raft_consensus.cc:1240] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:44.367287 12745 raft_consensus.cc:1275] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260504 20:01:44.364099 12742 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:44.370832 15535 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:44.371682 15364 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:44.375109 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:44.377250 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.379961 12821 raft_consensus.cc:1275] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:44.386266 12863 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:44.388907 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.385046 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:44.394723 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.395841 12894 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 8: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:44.396338 12895 raft_consensus.cc:2393] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:44.397984 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:44.294982 15380 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:44.331960 15635 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:44.405754 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:44.409785 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.409747 13497 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:44.412671 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:44.414222 13497 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:44.411896 12750 raft_consensus.cc:2410] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:01:44.400512 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.418994 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.420333 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:44.421470 12716 leader_election.cc:336] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.773s (SENT)
I20260504 20:01:44.423483 12716 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:44.413386 12749 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:44.422642 13497 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:44.429505 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.426977 15345 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:44.434120 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:44.434805 12816 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:44.431941 15638 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:44.436194 15613 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:44.436650 12897 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260504 20:01:44.437088 12897 pending_rounds.cc:85] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 4
I20260504 20:01:44.437355 12897 pending_rounds.cc:107] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260504 20:01:44.436195 15638 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:44.438161 15638 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:44.437595 12897 raft_consensus.cc:2889] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260504 20:01:44.447945 15655 log.cc:927] Time spent T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
W20260504 20:01:44.448812 15657 log.cc:927] Time spent T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.151s user 0.000s sys 0.001s
I20260504 20:01:44.076202 13530 rpcz_store.cc:276] 0504 20:01:42.960542 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:42.983814 (+ 23272us) service_pool.cc:224] Handling call
0504 20:01:43.823703 (+839889us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:43.839459 (+ 15756us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:43.842689 (+ 3230us) raft_consensus.cc:1534] Early marking committed up to index 8
0504 20:01:43.869026 (+ 26337us) log.cc:844] Serialized 14 byte log entry
0504 20:01:44.018009 (+148983us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:44.018068 (+ 59us) raft_consensus.cc:1645] Marking committed up to 8
0504 20:01:44.018136 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:44.018227 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:44.018351 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":5513,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":41,"raft.queue_time_us":8964,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"spinlock_wait_cycles":46225152,"thread_start_us":116936,"threads_started":1,"wal-append.queue_time_us":294902}
I20260504 20:01:44.458307 15605 consensus_queue.cc:1048] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:44.402925 15386 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260504 20:01:44.384665 15534 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260504 20:01:44.486477 13532 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 348) took 1892 ms. Trace:
I20260504 20:01:44.487164 13532 rpcz_store.cc:276] 0504 20:01:42.594390 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:42.594569 (+ 179us) service_pool.cc:224] Handling call
0504 20:01:42.594931 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:43.677559 (+1082628us) spinlock_profiling.cc:243] Waited 7.68 ms on lock 0x7b540051af20. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9261a0 00007f015c9b05a0 00007f015c9b1611 00007f015c9ae16f 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c
0504 20:01:43.681136 (+ 3577us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:44.448660 (+767524us) spinlock_profiling.cc:243] Waited 77.5 ms on lock 0x7b540051af20. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9261a0 00007f015c9b1de2 00007f015c9ae1b6 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22
0504 20:01:44.486073 (+ 37413us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b540051ac54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:44.486380 (+ 307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1911667584}
W20260504 20:01:44.520128 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.441988 15372 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:44.551693 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:44.552973 12898 raft_consensus.cc:2410] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260504 20:01:44.449101 15656 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
W20260504 20:01:44.561184 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 139720us
I20260504 20:01:44.442456 15560 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:44.566164 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.443120 15580 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election won for term 8
W20260504 20:01:44.580372 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 100305us
I20260504 20:01:44.584363 12895 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:44.587570 12742 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:01:44.589920 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:44.590619 12895 raft_consensus.cc:2376] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5. Re-sending same reply.
I20260504 20:01:44.587435 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.602799 12816 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:44.657764 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.442898 15501 consensus_queue.cc:1048] T c9a8aee69ec8499895f3ab719968d204 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.146s
W20260504 20:01:44.666065 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:44.720171 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 323137us
W20260504 20:01:44.759619 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 333270us
I20260504 20:01:44.835391 13497 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:44.765921 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:44.658350 15535 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:44.852348 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.563652 15364 consensus_queue.cc:1059] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.178s
I20260504 20:01:44.828861 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:44.829094 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 172041us
W20260504 20:01:44.855235 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:44.855841 12716 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.750s (SENT) [suppressed 34 similar messages]
I20260504 20:01:44.420917 15434 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.141s
I20260504 20:01:44.842269 12895 raft_consensus.cc:2410] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:44.856035 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:44.862550 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:44.865247 12791 leader_election.cc:336] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.821s (SENT)
I20260504 20:01:44.866120 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:44.868141 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.869443 12895 raft_consensus.cc:2410] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260504 20:01:44.870244 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:44.860706 15636 consensus_meta.cc:287] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.316s user 0.026s sys 0.041s
W20260504 20:01:44.882339 15661 log.cc:927] Time spent T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.205s user 0.001s sys 0.000s
I20260504 20:01:44.893335 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.894840 12895 raft_consensus.cc:2393] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:44.854836 12898 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
W20260504 20:01:44.904778 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.882736 15669 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:01:44.912016 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.916556 12895 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:44.918723 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.923220 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.928369 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.933198 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.937789 12895 raft_consensus.cc:2410] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:44.939900 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.940717 12898 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
W20260504 20:01:44.942569 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 86062.3us
I20260504 20:01:44.887876 15537 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:44.889247 15380 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:44.945701 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.946767 12895 raft_consensus.cc:2393] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 3: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:44.899355 15635 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:44.968432 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.883833 15651 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
I20260504 20:01:44.878094 15636 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:44.971063 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:44.978467 12895 raft_consensus.cc:2368] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8.
I20260504 20:01:44.979756 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.979974 15636 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:44.981118 12741 raft_consensus.cc:2368] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8.
I20260504 20:01:44.981202 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:44.982194 12791 leader_election.cc:336] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 3.019s (SENT)
I20260504 20:01:44.982685 12791 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:44.971503 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:44.972604 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:44.984952 12891 raft_consensus.cc:2368] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4.
I20260504 20:01:44.986706 12898 raft_consensus.cc:2368] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3.
W20260504 20:01:44.987677 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.987900 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.988415 15635 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:44.989056 12898 raft_consensus.cc:2410] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:44.988977 15635 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:44.968537 15660 log.cc:927] Time spent T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260504 20:01:44.992887 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:44.997081 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:44.977953 15380 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:45.001027 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:44.994725 15566 consensus_meta.cc:287] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.936s user 0.015s sys 0.028s
I20260504 20:01:45.004211 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.019927 12750 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:45.006522 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.021636 12745 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:45.001649 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.010356 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.023281 12891 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:44.994369 12714 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:45.009303 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:44.995141 15669 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:45.028084 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.035769 12746 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:45.028805 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:45.029546 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.036803 12747 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:45.030629 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:44.998546 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:45.043130 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 187311us
I20260504 20:01:45.033975 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.034423 13528 raft_consensus.cc:1240] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
W20260504 20:01:45.034356 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 199048us
I20260504 20:01:45.036394 12742 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:45.038316 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.045109 12891 raft_consensus.cc:2376] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5. Re-sending same reply.
I20260504 20:01:45.041410 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:45.045589 12748 raft_consensus.cc:1275] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:45.044492 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:45.045148 12895 raft_consensus.cc:1240] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:45.046411 12822 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:45.032855 12749 raft_consensus.cc:1240] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:01:45.047714 12895 raft_consensus.cc:1275] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260504 20:01:45.053167 13497 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:45.063591 12791 connection.cc:254] RPC call timeout handler was delayed by -2.6447817700000087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
W20260504 20:01:45.064211 12791 connection.cc:254] RPC call timeout handler was delayed by -2.63938177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
I20260504 20:01:45.029428 15535 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:45.064595 12791 connection.cc:254] RPC call timeout handler was delayed by -2.4827817699999741s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
W20260504 20:01:45.065026 12791 connection.cc:254] RPC call timeout handler was delayed by -2.4413817700000209s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
I20260504 20:01:45.065214 12745 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
W20260504 20:01:45.069980 12791 connection.cc:254] RPC call timeout handler was delayed by -2.392781770000056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1847s for a response
W20260504 20:01:45.070556 12791 connection.cc:254] RPC call timeout handler was delayed by -2.30638177000003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
I20260504 20:01:45.033840 15501 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:45.077698 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:45.081634 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:45.082610 12749 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:45.084959 12899 raft_consensus.cc:1275] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260504 20:01:45.085392 12899 pending_rounds.cc:85] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 2
I20260504 20:01:45.085611 12899 pending_rounds.cc:107] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260504 20:01:45.085851 12899 raft_consensus.cc:2889] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:45.087621 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:45.088059 12791 connection.cc:254] RPC call timeout handler was delayed by -2.2685817699999689s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20260504 20:01:45.088322 12748 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
W20260504 20:01:45.088483 12791 connection.cc:254] RPC call timeout handler was delayed by -1.7807817699999759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25270000000000004s for a response
W20260504 20:01:45.088835 12791 connection.cc:254] RPC call timeout handler was delayed by -1.6916817700000593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2626s for a response
W20260504 20:01:45.089135 12791 connection.cc:254] RPC call timeout handler was delayed by -1.4117817700000614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2937s for a response
W20260504 20:01:45.089560 12791 connection.cc:254] RPC call timeout handler was delayed by -1.4117817700000614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2937s for a response
W20260504 20:01:45.089855 12791 connection.cc:254] RPC call timeout handler was delayed by -1.3901817700000265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2961s for a response
W20260504 20:01:45.090164 12791 connection.cc:254] RPC call timeout handler was delayed by -1.1462817699999732s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32320000000000004s for a response
I20260504 20:01:45.094722 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.095435 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.095885 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 1992) took 1067 ms. Trace:
I20260504 20:01:45.052016 13530 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:45.096763 12749 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:45.098541 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.107525 12742 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:01:45.110072 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 244044us
I20260504 20:01:45.055689 15364 consensus_queue.cc:1243] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer ef6057d9a49e4942b47ea0fcd66ab6f7 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260504 20:01:45.096434 12817 rpcz_store.cc:276] 0504 20:01:44.028364 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.028496 (+ 132us) service_pool.cc:224] Handling call
0504 20:01:44.028816 (+ 320us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:44.029169 (+ 353us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:44.029197 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:44.029646 (+ 449us) log.cc:844] Serialized 26 byte log entry
0504 20:01:44.339197 (+309551us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:44.339242 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:44.339321 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:45.095674 (+756353us) raft_consensus.cc:1685] finished
0504 20:01:45.095695 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:45.095822 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2860,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":2248,"thread_start_us":309116,"threads_started":1,"wal-append.queue_time_us":576487}
I20260504 20:01:45.119681 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.120651 12899 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:45.127965 12891 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: []
W20260504 20:01:45.130169 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 79558.4us
I20260504 20:01:45.132274 12745 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:45.051388 15179 consensus_queue.cc:1059] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260504 20:01:45.148617 12863 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:45.157145 12822 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:01:45.096406 12750 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:45.121002 15580 raft_consensus.cc:697] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:45.170002 12714 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:45.164858 12716 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:45.091899 15535 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:45.180828 15675 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:45.186301 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.187072 12747 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:45.188149 15636 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:01:45.188808 15636 raft_consensus.cc:697] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:45.174029 15179 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:45.189791 15636 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:45.181600 15666 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
I20260504 20:01:45.193619 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 479) took 1125 ms. Trace:
I20260504 20:01:45.130074 15669 consensus_queue.cc:1048] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260504 20:01:45.194178 12893 rpcz_store.cc:276] 0504 20:01:44.068085 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.068343 (+ 258us) service_pool.cc:224] Handling call
0504 20:01:44.152526 (+ 84183us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:44.166410 (+ 13884us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:44.166459 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:44.186433 (+ 19974us) log.cc:844] Serialized 52 byte log entry
0504 20:01:44.581024 (+394591us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:44.651487 (+ 70463us) log.cc:844] Serialized 14 byte log entry
0504 20:01:44.833611 (+182124us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:44.833722 (+ 111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:45.193321 (+359599us) raft_consensus.cc:1685] finished
0504 20:01:45.193356 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:45.193538 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":180978,"prepare.queue_time_us":81640,"prepare.run_cpu_time_us":6257,"prepare.run_wall_time_us":85742,"raft.queue_time_us":227646,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":7776,"spinlock_wait_cycles":10570112,"thread_start_us":292577,"threads_started":1,"wal-append.queue_time_us":642916}
I20260504 20:01:45.173138 12742 raft_consensus.cc:2410] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260504 20:01:45.204705 12747 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:45.205773 12746 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:45.214619 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:45.216224 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 57154.9us
I20260504 20:01:45.217458 12716 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:45.218834 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.219756 12747 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:01:45.221567 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 168501us
I20260504 20:01:45.224314 12746 raft_consensus.cc:2410] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:45.234774 12893 raft_consensus.cc:1275] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260504 20:01:45.246338 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:45.248034 13527 raft_consensus.cc:1240] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260504 20:01:45.252014 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:45.257030 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:45.261716 12746 raft_consensus.cc:1275] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:45.214542 15501 raft_consensus.cc:697] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:45.272320 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.273819 12746 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:45.282763 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.283799 12750 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:01:45.296657 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 139887us
W20260504 20:01:45.297183 12820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 1989) took 1362 ms (client timeout 1703 ms). Trace:
W20260504 20:01:45.297646 12820 rpcz_store.cc:269] 0504 20:01:43.909029 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:43.909118 (+ 89us) service_pool.cc:224] Handling call
0504 20:01:45.256784 (+1347666us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:45.270933 (+ 14149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3178752}
I20260504 20:01:45.298585 12747 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
W20260504 20:01:45.301271 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 146723us
I20260504 20:01:45.251708 15651 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260504 20:01:45.334101 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:45.336301 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 109223us
I20260504 20:01:45.357435 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:45.298003 15635 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:45.375756 12746 raft_consensus.cc:2410] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260504 20:01:45.381361 12750 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
W20260504 20:01:45.384140 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:45.386620 15483 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:45.391496 12748 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:01:45.396044 13497 leader_election.cc:336] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.717s (SENT)
I20260504 20:01:45.396616 13524 raft_consensus.cc:1217] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 LEADER]: Deduplicated request from leader. Original: 3.6->[4.7-4.7] Dedup: 4.7->[]
I20260504 20:01:45.397169 13524 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260504 20:01:45.342049 15690 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:45.375499 15534 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:45.413167 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 75361us
I20260504 20:01:45.413883 12716 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:45.423391 12716 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:45.296108 15580 consensus_queue.cc:237] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:45.428797 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:45.416625 15643 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:45.427160 12716 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:45.397233 15681 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:45.437306 12865 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:45.440435 12748 raft_consensus.cc:1275] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260504 20:01:45.450292 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 152817us
I20260504 20:01:45.389489 15483 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.024s
I20260504 20:01:45.453475 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:45.460999 12716 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:45.440102 15690 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:45.462860 15685 log.cc:927] Time spent T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260504 20:01:45.464301 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1960) took 1115 ms. Trace:
I20260504 20:01:45.464740 12818 rpcz_store.cc:276] 0504 20:01:44.349141 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.349339 (+ 198us) service_pool.cc:224] Handling call
0504 20:01:44.350318 (+ 979us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:44.658812 (+308494us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:44.658844 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:44.675680 (+ 16836us) log.cc:844] Serialized 26 byte log entry
0504 20:01:45.098420 (+422740us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:45.098461 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:45.098533 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:45.464002 (+365469us) raft_consensus.cc:1685] finished
0504 20:01:45.464023 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:45.464215 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113230,"raft.queue_time_us":281225,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":65056,"thread_start_us":483389,"threads_started":2,"wal-append.queue_time_us":389038}
I20260504 20:01:45.441562 15635 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:45.452831 12865 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:45.468595 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:45.478768 12750 raft_consensus.cc:1275] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:45.448050 15534 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:45.476719 15682 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:45.492761 15682 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:45.493343 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.494980 15682 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:45.473063 15651 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:45.476425 15432 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:45.496932 15648 consensus_meta.cc:287] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.343s user 0.019s sys 0.004s
W20260504 20:01:45.490916 15683 log.cc:927] Time spent T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260504 20:01:45.476521 15690 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:45.499253 12746 pending_rounds.cc:85] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting all ops after (but not including) 3
I20260504 20:01:45.491766 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:45.500231 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.513291 12748 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:45.425765 15501 consensus_queue.cc:237] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:45.503168 15696 consensus_queue.cc:1059] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:45.504614 12716 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:45.451560 15681 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:45.505849 15648 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:45.447568 15643 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:45.510284 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 297) took 1664 ms. Trace:
I20260504 20:01:45.505851 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:45.512564 12746 pending_rounds.cc:107] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting uncommitted NO_OP operation due to leader change: 6.4
W20260504 20:01:45.514765 15638 consensus_meta.cc:287] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.076s user 0.019s sys 0.014s
W20260504 20:01:45.515750 15345 consensus_meta.cc:287] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.083s user 0.011s sys 0.011s
I20260504 20:01:45.519688 12749 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:45.525558 12825 rpcz_store.cc:276] 0504 20:01:43.845548 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:43.914948 (+ 69400us) service_pool.cc:224] Handling call
0504 20:01:43.955100 (+ 40152us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:43.966623 (+ 11523us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:45.510046 (+1543423us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b54000e4cd0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:45.510216 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91562240}
I20260504 20:01:45.522359 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:45.498236 15179 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:45.526257 12746 raft_consensus.cc:2889] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:45.526052 12820 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:45.535208 12747 raft_consensus.cc:1275] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:45.521689 12741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1819) took 1155 ms. Trace:
I20260504 20:01:45.539901 12741 rpcz_store.cc:276] 0504 20:01:44.362218 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.384681 (+ 22463us) service_pool.cc:224] Handling call
0504 20:01:45.512078 (+1127397us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b540022fbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:45.517567 (+ 5489us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1349330304}
I20260504 20:01:45.528029 15432 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:45.527360 15542 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:45.541780 12863 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:45.542613 12716 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:45.530630 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 134203us
I20260504 20:01:45.546622 12744 raft_consensus.cc:2368] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9.
I20260504 20:01:45.525424 15651 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:45.527776 15693 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:45.560211 12748 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:45.541270 15345 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:45.572535 12750 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:45.588482 13532 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:45.553448 15364 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:45.589618 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:45.595943 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 53396.1us
I20260504 20:01:45.562294 15698 raft_consensus.cc:3055] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:45.557201 15179 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:45.597518 12716 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:45.538326 15638 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:45.574095 15534 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:45.612063 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 469) took 2159 ms. Trace:
I20260504 20:01:45.618387 12896 rpcz_store.cc:276] 0504 20:01:43.439675 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:43.439854 (+ 179us) service_pool.cc:224] Handling call
0504 20:01:43.467707 (+ 27853us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:44.172164 (+704457us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:45.598690 (+1426526us) spinlock_profiling.cc:243] Waited 689 ms on lock 0x7b54003b9a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:45.598938 (+ 248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1585145728}
I20260504 20:01:45.618923 12716 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:45.555785 15542 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:45.625334 12716 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:45.627133 12716 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:45.629642 13531 raft_consensus.cc:1275] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
W20260504 20:01:45.629722 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 86593.2us
I20260504 20:01:45.554855 15432 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:45.636831 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 64865.7us
I20260504 20:01:45.639424 12750 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:45.630172 13531 pending_rounds.cc:85] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Aborting all ops after (but not including) 5
I20260504 20:01:45.569029 15700 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260504 20:01:45.645306 13531 pending_rounds.cc:107] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260504 20:01:45.645658 13531 raft_consensus.cc:2889] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:45.606032 15179 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:45.605374 15698 raft_consensus.cc:740] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:45.686172 13531 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 630) took 1681 ms. Trace:
I20260504 20:01:45.599128 15364 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:45.633096 15650 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:45.643411 13533 raft_consensus.cc:1240] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:45.642437 15635 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:45.571606 15693 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:45.659082 12865 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:45.686757 13531 rpcz_store.cc:276] 0504 20:01:44.004800 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.004969 (+ 169us) service_pool.cc:224] Handling call
0504 20:01:45.628989 (+1624020us) spinlock_profiling.cc:243] Waited 936 us on lock 0x7b58000f3c98. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:01:45.629242 (+ 253us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:45.646459 (+ 17217us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:45.686074 (+ 39615us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2153344}
I20260504 20:01:45.598563 15654 consensus_queue.cc:1048] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.038s
I20260504 20:01:45.642792 15672 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:45.643047 15651 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:45.684139 15179 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:45.710510 15635 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:01:45.708768 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 49812.3us
W20260504 20:01:45.708303 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 95361.5us
I20260504 20:01:45.644066 13532 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
W20260504 20:01:45.648279 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 93462.6us
I20260504 20:01:45.774060 12894 raft_consensus.cc:1275] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:45.724330 15654 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
W20260504 20:01:45.785516 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 158501us
I20260504 20:01:45.786567 12716 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:45.798300 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:45.799237 12741 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 7: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:45.797554 15648 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:45.800981 15698 consensus_queue.cc:260] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:45.801662 15432 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:45.801862 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:45.802211 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:45.802608 12823 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:45.802748 13528 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Advancing to term 9
I20260504 20:01:45.803213 12716 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:45.764427 15651 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:45.780408 15635 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:45.818578 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:45.855104 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 52043.8us
W20260504 20:01:45.873090 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:45.818055 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 180250us
I20260504 20:01:45.873270 12716 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:45.875007 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:45.870765 13530 consensus_meta.cc:287] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.740s user 0.008s sys 0.004s
I20260504 20:01:45.852264 15650 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:45.870338 15651 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:45.872036 15702 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:45.869128 15672 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:45.798213 15682 consensus_queue.cc:1059] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 7, Time since last communication: 0.101s
I20260504 20:01:45.870407 15643 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:45.926509 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 386986us
I20260504 20:01:45.870232 15681 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:45.870872 15648 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:45.910355 13530 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:45.876761 15654 raft_consensus.cc:697] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:45.853646 15698 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:45.804543 15709 consensus_queue.cc:1048] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.130s
W20260504 20:01:45.974187 12891 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.009s sys 0.007s
W20260504 20:01:46.021196 12891 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.634s user 0.010s sys 0.007s
I20260504 20:01:45.997589 15701 consensus_queue.cc:1059] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260504 20:01:46.003155 15700 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:45.919507 15636 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:46.011178 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:46.017932 12820 raft_consensus.cc:1275] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:46.023207 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:46.013126 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 135035us
I20260504 20:01:46.093236 12892 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:45.853250 15716 consensus_queue.cc:1059] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.197s
I20260504 20:01:46.013228 13528 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
I20260504 20:01:46.055264 12823 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
W20260504 20:01:46.022583 15666 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
W20260504 20:01:46.099256 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 320918us
I20260504 20:01:46.099577 12791 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:46.003423 15534 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:46.023555 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 633) took 1050 ms. Trace:
W20260504 20:01:46.093943 12892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1947) took 1978 ms (client timeout 1667 ms). Trace:
W20260504 20:01:46.111833 12892 rpcz_store.cc:269] 0504 20:01:44.115088 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.135758 (+ 20670us) service_pool.cc:224] Handling call
0504 20:01:46.023022 (+1887264us) spinlock_profiling.cc:243] Waited 7.83 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:01:46.093797 (+ 70775us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17997568}
W20260504 20:01:46.063153 15685 log.cc:927] Time spent T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260504 20:01:46.045511 15638 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:46.104844 12791 leader_election.cc:400] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8.
I20260504 20:01:46.125157 12791 leader_election.cc:403] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260504 20:01:46.006074 15648 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.102691 15730 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:46.109689 15483 consensus_queue.cc:1048] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:01:46.110590 15701 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:46.058096 15700 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:46.064160 15542 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:46.114197 13531 raft_consensus.cc:1240] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260504 20:01:46.004590 15704 consensus_queue.cc:1059] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260504 20:01:46.107178 15716 consensus_queue.cc:1048] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:45.962352 15702 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.015079 15681 consensus_queue.cc:1059] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 7, Time since last communication: 0.086s
W20260504 20:01:46.129243 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 254336us
I20260504 20:01:46.172163 12716 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:46.131847 12748 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260504 20:01:46.135991 12791 leader_election.cc:336] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.747s (SENT)
I20260504 20:01:46.001195 15672 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.143191 12741 raft_consensus.cc:1240] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260504 20:01:46.144194 12749 raft_consensus.cc:1240] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260504 20:01:46.111622 13530 rpcz_store.cc:276] 0504 20:01:44.968738 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.973195 (+ 4457us) service_pool.cc:224] Handling call
0504 20:01:46.018728 (+1045533us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:46.152289 12747 raft_consensus.cc:1240] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260504 20:01:46.158267 15700 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:46.147253 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 201198us
I20260504 20:01:46.085155 15721 consensus_queue.cc:1048] T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.097s
W20260504 20:01:46.173878 15705 log.cc:927] Time spent T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.282s user 0.000s sys 0.001s
I20260504 20:01:46.151218 15627 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:46.177639 12716 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:46.133901 15364 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:46.155009 15722 raft_consensus.cc:3055] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Stepping down as leader of term 9
W20260504 20:01:46.175979 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 404792us
I20260504 20:01:46.181170 13531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:46.192992 12865 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:46.176566 12749 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8. Ops: []
I20260504 20:01:46.187027 12714 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:46.151512 15372 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260504 20:01:46.195257 15717 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:01:46.195785 15717 raft_consensus.cc:697] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:46.196821 15724 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007) with higher term. Message: Invalid argument: T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8.
I20260504 20:01:46.196705 15717 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.198894 15702 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:46.160808 15613 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260504 20:01:46.192265 15722 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:46.203747 15722 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.205133 15722 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:01:46.192479 15636 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260504 20:01:46.208222 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 72279.9us
I20260504 20:01:46.208451 12716 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:46.176755 15681 consensus_queue.cc:1243] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer ef6057d9a49e4942b47ea0fcd66ab6f7 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260504 20:01:46.216708 12791 leader_election.cc:336] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.793s (SENT)
I20260504 20:01:46.218652 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:46.188148 15736 raft_consensus.cc:3055] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:46.222384 12747 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:46.189666 15720 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:46.185678 15693 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:46.229346 12865 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:46.230465 15695 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:46.260571 15695 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.261189 15648 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:46.261996 15745 consensus_queue.cc:1048] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:46.263495 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 632) took 2158 ms. Trace:
I20260504 20:01:46.263893 13527 rpcz_store.cc:276] 0504 20:01:44.105394 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.105560 (+ 166us) service_pool.cc:224] Handling call
0504 20:01:44.134437 (+ 28877us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:45.250238 (+1115801us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:46.263180 (+1012942us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54004b0354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:46.263406 (+ 226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2560377088}
I20260504 20:01:46.147876 15730 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.230989 15684 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:01:46.167590 15654 consensus_queue.cc:237] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.233696 15710 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:46.201449 15627 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.230792 15726 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:46.240622 15651 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.222343 15736 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:46.250466 15009 raft_consensus.cc:3055] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:46.257332 12716 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:46.260392 15642 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.230696 15731 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:46.301182 15009 raft_consensus.cc:740] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:46.322041 15726 raft_consensus.cc:697] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:46.323041 15672 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:46.287468 15684 raft_consensus.cc:697] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:46.323262 15726 consensus_queue.cc:237] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.269443 15432 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260504 20:01:46.324584 15684 consensus_queue.cc:237] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.325146 15627 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:46.271360 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:46.270186 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 125226us
I20260504 20:01:46.248474 15179 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.288071 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:46.303192 15636 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:46.273943 15534 consensus_queue.cc:1048] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260504 20:01:46.305558 15736 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.323000 15009 consensus_queue.cc:260] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.269937 12865 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:46.351316 15009 raft_consensus.cc:3060] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:46.338502 12748 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:46.271313 15695 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:46.355692 15741 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:46.356431 15741 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:46.360383 12743 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57934 (request call id 409) took 2368 ms (client timeout 1928 ms). Trace:
W20260504 20:01:46.360864 12743 rpcz_store.cc:269] 0504 20:01:43.992286 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.003005 (+ 10719us) service_pool.cc:224] Handling call
0504 20:01:46.360149 (+2357144us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54003da254. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:46.360300 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432297728}
I20260504 20:01:46.362102 12742 raft_consensus.cc:1240] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:46.286762 15710 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:46.360235 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:46.267818 15730 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:46.458505 12895 raft_consensus.cc:3055] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
W20260504 20:01:46.405362 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 174028us
I20260504 20:01:46.552930 12895 raft_consensus.cc:740] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:46.553507 12791 leader_election.cc:336] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.985s (SENT)
I20260504 20:01:46.554100 12791 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:46.456863 13532 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 634) took 1456 ms (client timeout 1591 ms). Trace:
W20260504 20:01:46.555848 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 258053us
I20260504 20:01:46.487528 12818 raft_consensus.cc:2410] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
W20260504 20:01:46.364966 12890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1924) took 2395 ms (client timeout 1975 ms). Trace:
W20260504 20:01:46.557605 12890 rpcz_store.cc:269] 0504 20:01:43.969688 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:43.970439 (+ 751us) service_pool.cc:224] Handling call
0504 20:01:46.364734 (+2394295us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54000109d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:46.364884 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":711739264}
W20260504 20:01:46.550801 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 293574us
W20260504 20:01:46.559223 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 412377us
I20260504 20:01:46.559391 12716 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:46.562522 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:46.436708 12862 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.006s sys 0.017s
W20260504 20:01:46.556123 13532 rpcz_store.cc:269] 0504 20:01:45.000509 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:45.000728 (+ 219us) service_pool.cc:224] Handling call
0504 20:01:46.456664 (+1455936us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54005be354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:46.456788 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1330849536}
W20260504 20:01:46.565420 12862 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s user 0.009s sys 0.017s
I20260504 20:01:46.368146 12898 raft_consensus.cc:1240] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:44.834806 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:01:46.568794 15483 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:46.591967 12716 connection.cc:254] RPC call timeout handler was delayed by -1.6571828829999049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16670000000000001s for a response
I20260504 20:01:46.591276 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:46.594296 12716 connection.cc:254] RPC call timeout handler was delayed by -1.142382882999982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2239s for a response
I20260504 20:01:46.554595 12895 consensus_queue.cc:260] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.599418 12895 raft_consensus.cc:3060] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:46.484094 15718 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.388s user 0.000s sys 0.001s
W20260504 20:01:46.541913 15706 log.cc:927] Time spent T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.817s user 0.002s sys 0.003s
I20260504 20:01:46.596997 15650 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:46.539404 15707 log.cc:927] Time spent T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.732s user 0.001s sys 0.001s
W20260504 20:01:46.614837 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 61443.3us
W20260504 20:01:46.615849 12791 leader_election.cc:336] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.527s (SENT)
I20260504 20:01:46.616382 12791 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:46.458596 13525 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260504 20:01:46.621138 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:46.455502 15535 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:46.627080 12744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1822) took 2263 ms (client timeout 1711 ms). Trace:
I20260504 20:01:46.627274 12895 raft_consensus.cc:1275] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260504 20:01:46.627923 12895 pending_rounds.cc:85] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 6
I20260504 20:01:46.454133 15179 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:46.627636 12744 rpcz_store.cc:269] 0504 20:01:44.363347 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.364579 (+ 1232us) service_pool.cc:224] Handling call
0504 20:01:46.626833 (+2262254us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54000dc0d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:46.626993 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2715772160}
I20260504 20:01:46.629410 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:46.630136 15483 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:46.645624 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:46.647396 13530 raft_consensus.cc:2410] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260504 20:01:46.637257 15710 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:46.628381 12895 pending_rounds.cc:107] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260504 20:01:46.651221 12895 raft_consensus.cc:2889] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:46.652454 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 481) took 2464 ms. Trace:
I20260504 20:01:46.362520 15741 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:46.597856 15534 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:46.619460 15543 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.307s user 0.012s sys 0.059s
I20260504 20:01:46.674786 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:46.687379 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 71579.5us
I20260504 20:01:46.669701 15651 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:46.652935 12895 rpcz_store.cc:276] 0504 20:01:44.187741 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:45.049925 (+862184us) service_pool.cc:224] Handling call
0504 20:01:45.050196 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:46.458397 (+1408201us) spinlock_profiling.cc:243] Waited 70.6 ms on lock 0x7b54002952a0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9261a0 00007f015c9b05a0 00007f015c9b1611 00007f015c9ae16f 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c
0504 20:01:46.652089 (+193692us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:46.652378 (+ 289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162351872}
W20260504 20:01:46.680837 15543 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.375s user 0.012s sys 0.059s
I20260504 20:01:46.675701 15483 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:46.699275 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1970) took 1214 ms. Trace:
I20260504 20:01:46.700723 12893 rpcz_store.cc:276] 0504 20:01:45.484529 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:45.494300 (+ 9771us) service_pool.cc:224] Handling call
0504 20:01:45.494632 (+ 332us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:45.495014 (+ 382us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:45.495046 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:45.495607 (+ 561us) log.cc:844] Serialized 26 byte log entry
0504 20:01:45.591035 (+ 95428us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:46.044728 (+453693us) log.cc:844] Serialized 14 byte log entry
0504 20:01:46.044975 (+ 247us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:46.045076 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:46.540877 (+495801us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:46.698814 (+157937us) raft_consensus.cc:1685] finished
0504 20:01:46.698955 (+ 141us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:46.699170 (+ 215us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":406388,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":62609,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":219,"spinlock_wait_cycles":31149440,"thread_start_us":507138,"threads_started":2,"wal-append.queue_time_us":136724}
W20260504 20:01:46.710412 12791 leader_election.cc:336] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.961s (SENT)
I20260504 20:01:46.712740 15698 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:46.713789 13533 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 360) took 1734 ms. Trace:
I20260504 20:01:46.713474 15698 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.714308 15483 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:46.716462 15623 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260504 20:01:46.714301 13533 rpcz_store.cc:276] 0504 20:01:44.979785 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.979931 (+ 146us) service_pool.cc:224] Handling call
0504 20:01:44.980223 (+ 292us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:45.702946 (+722723us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:46.713492 (+1010546us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54004b1254. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:46.713715 (+ 223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1524681088}
W20260504 20:01:46.707329 15635 consensus_meta.cc:287] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.788s user 0.011s sys 0.002s
I20260504 20:01:46.680536 15534 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.727772 15635 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.718035 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:46.729354 15700 consensus_meta.cc:287] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.539s user 0.007s sys 0.005s
W20260504 20:01:46.747851 15723 log.cc:927] Time spent T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.639s user 0.000s sys 0.002s
W20260504 20:01:46.756542 12716 leader_election.cc:336] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.667s (SENT)
I20260504 20:01:46.757114 12716 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:46.758128 15651 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:46.759773 15694 log.cc:927] Time spent T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 1.163s user 0.008s sys 0.000s
I20260504 20:01:46.761054 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:46.761262 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:46.762079 12745 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
W20260504 20:01:46.762662 12745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1710) took 1577 ms (client timeout 1882 ms). Trace:
I20260504 20:01:46.762569 12899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1746) took 1688 ms. Trace:
W20260504 20:01:46.763155 12745 rpcz_store.cc:269] 0504 20:01:45.185188 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:46.760746 (+1575558us) service_pool.cc:224] Handling call
0504 20:01:46.762600 (+ 1854us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:46.763243 12899 rpcz_store.cc:276] 0504 20:01:45.074682 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:45.160389 (+ 85707us) service_pool.cc:224] Handling call
0504 20:01:45.179528 (+ 19139us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:45.179772 (+ 244us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:45.179800 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:45.180653 (+ 853us) log.cc:844] Serialized 26 byte log entry
0504 20:01:45.261950 (+ 81297us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:45.294529 (+ 32579us) log.cc:844] Serialized 14 byte log entry
0504 20:01:45.317376 (+ 22847us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:45.317481 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:46.488642 (+1171161us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b18001473a0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159910b62 00007f0159912e01 00007f0159912db9 00007f0159912d49 00007f0159912d11 00007f015991200d 0000000000379254 0000000000379099 00007f01598f3fba
0504 20:01:46.717906 (+229264us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:46.762311 (+ 44405us) raft_consensus.cc:1685] finished
0504 20:01:46.762344 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:46.762504 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32535,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":1762,"raft.queue_time_us":11355,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":34138,"spinlock_wait_cycles":59337728,"thread_start_us":68328,"threads_started":1,"wal-append.queue_time_us":370565}
I20260504 20:01:46.744387 15483 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:46.774076 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1719) took 1279 ms. Trace:
I20260504 20:01:46.774611 12746 rpcz_store.cc:276] 0504 20:01:45.494921 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:45.498666 (+ 3745us) service_pool.cc:224] Handling call
0504 20:01:45.499021 (+ 355us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:45.643434 (+144413us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:45.643479 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:45.644514 (+ 1035us) log.cc:844] Serialized 52 byte log entry
0504 20:01:45.804529 (+160015us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:45.876429 (+ 71900us) log.cc:844] Serialized 14 byte log entry
0504 20:01:46.013601 (+137172us) log.cc:844] Serialized 14 byte log entry
0504 20:01:46.013892 (+ 291us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:46.014010 (+ 118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:46.723847 (+709837us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:46.773827 (+ 49980us) raft_consensus.cc:1685] finished
0504 20:01:46.773853 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:46.774005 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34674,"prepare.queue_time_us":107987,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":15906,"raft.queue_time_us":292079,"raft.run_cpu_time_us":553,"raft.run_wall_time_us":57560,"spinlock_wait_cycles":37413120,"thread_start_us":323167,"threads_started":3,"wal-append.queue_time_us":424573}
I20260504 20:01:46.776072 15439 raft_consensus.cc:3055] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:46.779695 15439 raft_consensus.cc:740] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:46.744031 15700 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.790032 13529 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260504 20:01:46.791149 15700 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:46.781852 15534 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:46.792508 15700 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260504 20:01:46.786195 15736 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:46.788779 12897 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1954) took 2615 ms (client timeout 2239 ms). Trace:
W20260504 20:01:46.783785 15737 log.cc:927] Time spent T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.673s user 0.000s sys 0.002s
I20260504 20:01:46.793390 15635 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:46.798728 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:46.805512 12897 rpcz_store.cc:269] 0504 20:01:44.173903 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.491608 (+317705us) service_pool.cc:224] Handling call
0504 20:01:46.788550 (+2296942us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b5400208ad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:46.788697 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2278415616}
W20260504 20:01:46.821923 12716 leader_election.cc:336] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.239s (SENT)
I20260504 20:01:46.822517 12716 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:46.823004 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:46.823843 12746 raft_consensus.cc:2376] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4. Re-sending same reply.
I20260504 20:01:46.827623 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:46.838209 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:46.842387 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:46.790311 15439 consensus_queue.cc:260] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.873482 13499 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:46.841395 15650 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:46.882475 12823 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260504 20:01:46.877187 15439 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:46.879603 15674 raft_consensus.cc:2804] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:46.900381 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:46.904003 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:46.908537 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:46.924394 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:46.902819 15674 raft_consensus.cc:697] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:46.926000 13524 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: []
I20260504 20:01:46.946707 12746 pending_rounds.cc:85] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting all ops after (but not including) 5
I20260504 20:01:46.947135 12746 pending_rounds.cc:107] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260504 20:01:46.947432 12746 raft_consensus.cc:2889] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:46.924396 15696 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:46.940641 15668 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:46.950536 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:46.931422 15710 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260504 20:01:46.959877 15674 consensus_queue.cc:237] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.954294 15696 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:46.981328 12825 raft_consensus.cc:2410] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260504 20:01:46.955224 15668 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:46.984683 15650 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260504 20:01:46.987432 12714 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:46.990406 12823 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260504 20:01:46.994786 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1966) took 2339 ms. Trace:
I20260504 20:01:46.999991 15696 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:46.995191 15690 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260504 20:01:47.004168 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:46.994604 15672 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260504 20:01:47.010690 15668 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.015842 15698 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.016942 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1732) took 2050 ms. Trace:
I20260504 20:01:47.017385 12898 rpcz_store.cc:276] 0504 20:01:44.966643 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:45.065739 (+ 99096us) service_pool.cc:224] Handling call
0504 20:01:46.367867 (+1302128us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:46.566230 (+198363us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:47.016715 (+450485us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54000109d0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:47.016859 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2993783680}
I20260504 20:01:46.995570 12824 rpcz_store.cc:276] 0504 20:01:44.650972 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:44.656172 (+ 5200us) service_pool.cc:224] Handling call
0504 20:01:44.656582 (+ 410us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:44.856608 (+200026us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:44.856643 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:44.874302 (+ 17659us) log.cc:844] Serialized 26 byte log entry
0504 20:01:46.779224 (+1904922us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b1800464fe0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d471 00007f015c969def
0504 20:01:46.911728 (+132504us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:46.951395 (+ 39667us) log.cc:844] Serialized 14 byte log entry
0504 20:01:46.954571 (+ 3176us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:46.955405 (+ 834us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:46.986815 (+ 31410us) raft_consensus.cc:1685] finished
0504 20:01:46.989911 (+ 3096us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:46.990064 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":186409,"prepare.queue_time_us":62702,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":221728,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":26627,"spinlock_wait_cycles":3183872,"thread_start_us":166099,"threads_started":2,"wal-append.queue_time_us":2059111}
I20260504 20:01:47.022548 15717 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:47.023088 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:47.023061 15717 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:47.024835 13524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 437) took 1584 ms. Trace:
I20260504 20:01:47.025355 13524 rpcz_store.cc:276] 0504 20:01:45.440973 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:45.441164 (+ 191us) service_pool.cc:224] Handling call
0504 20:01:45.441608 (+ 444us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:46.928339 (+1486731us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:47.021749 (+ 93410us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b540051d1d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:47.024740 (+ 2991us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3412969728}
I20260504 20:01:47.029600 15743 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:47.030645 15439 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:47.030901 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 761024us
I20260504 20:01:47.031433 15439 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:47.043964 15439 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.051964 15749 consensus_queue.cc:1048] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260504 20:01:47.052874 13526 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:01:47.052059 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:47.056978 15690 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260504 20:01:47.058051 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:47.058205 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1711) took 1872 ms. Trace:
I20260504 20:01:47.058909 15536 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:01:47.060168 15536 raft_consensus.cc:697] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:47.061185 15536 consensus_queue.cc:237] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.066140 15542 consensus_queue.cc:1059] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:01:47.066807 15668 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:47.067306 15542 raft_consensus.cc:3055] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:01:47.067673 15542 raft_consensus.cc:740] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:47.067492 15668 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.068552 15542 consensus_queue.cc:260] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:47.069530 15627 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:47.069754 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:47.070147 15627 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:47.072337 15627 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:47.059151 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 488242us
I20260504 20:01:47.073089 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:47.075425 15722 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:47.088047 15722 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.075891 15699 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:47.075906 12892 raft_consensus.cc:1240] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260504 20:01:47.090404 15699 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:47.069571 15668 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.091888 15668 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:47.092526 15699 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:47.092470 15668 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:47.093132 15699 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:47.076423 15736 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:47.077756 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:47.059629 15605 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:47.095186 15736 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.095726 15542 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:47.078387 12865 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:47.095883 15605 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:47.096753 15680 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260504 20:01:47.097220 15736 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.078809 12893 raft_consensus.cc:1240] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 7. Current term is 8. Ops: []
I20260504 20:01:47.058709 12742 rpcz_store.cc:276] 0504 20:01:45.185495 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:45.226885 (+ 41390us) service_pool.cc:224] Handling call
0504 20:01:46.361906 (+1135021us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:46.362555 (+ 649us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:47.057988 (+695433us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54003da250. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:47.058140 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2609423744}
I20260504 20:01:47.074750 15483 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:47.099435 15605 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:47.099277 15483 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.100590 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.101424 15483 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.082089 15623 consensus_queue.cc:1048] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:01:47.084609 15687 consensus_queue.cc:1048] T e44b757a449142a38ed04025e01cb4a5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:01:47.106034 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:47.086400 15693 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:47.081876 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:47.110699 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:47.111418 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.113255 13524 raft_consensus.cc:2393] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:47.116267 12825 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:47.145196 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:47.149210 15717 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:47.153877 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 49664.5us
I20260504 20:01:47.157960 12865 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:47.159106 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.159883 12825 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:01:47.112571 15743 consensus_queue.cc:1048] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260504 20:01:47.164713 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.155580 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:47.168546 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:47.167972 15680 consensus_queue.cc:1048] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:47.174700 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:47.175441 12818 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:47.093650 15699 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.138608 15672 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:47.188462 12816 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:01:47.180014 12865 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:47.106246 15380 consensus_queue.cc:1059] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260504 20:01:47.192106 12825 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:47.195554 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.203145 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:47.209897 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:47.212743 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:47.213549 12745 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:47.214268 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.215679 12816 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:47.217523 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:47.224200 12818 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:47.170390 15534 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260504 20:01:47.227021 12647 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39332 (request call id 224) took 16780 ms (client timeout 15000 ms). Trace:
I20260504 20:01:47.227336 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:47.228915 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:47.230020 15647 raft_consensus.cc:3055] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:01:47.230044 12742 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260504 20:01:47.240979 15699 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.241314 12863 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:47.243531 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:47.187968 15711 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:47.244635 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.245033 15668 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:01:47.162985 15605 consensus_queue.cc:1048] T 927c932a914241598bc58bcb0d84c2d4 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260504 20:01:47.161870 15693 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.256515 12825 raft_consensus.cc:1275] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:47.257098 12745 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:47.258853 12743 raft_consensus.cc:1275] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:01:47.275116 12746 raft_consensus.cc:1240] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:47.247152 15722 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.281076 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:47.212291 15623 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:47.293399 12749 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:01:47.294456 12863 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:47.303822 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.257761 15647 raft_consensus.cc:740] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:47.303853 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:47.250505 12647 rpcz_store.cc:269] 0504 20:01:30.446595 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:30.458193 (+ 11598us) service_pool.cc:224] Handling call
0504 20:01:47.226890 (+16768697us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:42.237559 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:42.358347 (+120788us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:42.358403 (+ 56us) write_op.cc:435] Acquired schema lock
0504 20:01:42.358430 (+ 27us) tablet.cc:662] Decoding operations
0504 20:01:42.359342 (+ 912us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:42.359422 (+ 80us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:42.359631 (+ 209us) tablet.cc:685] Acquiring locks for 7 operations
0504 20:01:42.360376 (+ 745us) tablet.cc:701] Row locks acquired
0504 20:01:42.360391 (+ 15us) write_op.cc:260] PREPARE: finished
0504 20:01:42.360593 (+ 202us) write_op.cc:270] Start()
0504 20:01:42.360724 (+ 131us) write_op.cc:276] Timestamp: P: 1777924902360524 usec, L: 0
0504 20:01:42.360759 (+ 35us) op_driver.cc:348] REPLICATION: starting
0504 20:01:42.439027 (+ 78268us) log.cc:844] Serialized 2808 byte log entry
0504 20:01:43.370594 (+931567us) op_driver.cc:464] REPLICATION: finished
0504 20:01:43.614564 (+243970us) write_op.cc:301] APPLY: starting
0504 20:01:43.614680 (+ 116us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:43.700944 (+ 86264us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:43.782894 (+ 81950us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:43.878986 (+ 96092us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:43.879381 (+ 395us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0504 20:01:43.879438 (+ 57us) write_op.cc:312] APPLY: finished
0504 20:01:43.879821 (+ 383us) log.cc:844] Serialized 59 byte log entry
0504 20:01:43.931467 (+ 51646us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:43.956914 (+ 25447us) write_op.cc:454] Released schema lock
0504 20:01:44.002735 (+ 45821us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:44.028721 (+ 25986us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:44.028752 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4,"reported_tablets":40,"child_traces":[["op",{"apply.queue_time_us":216046,"apply.run_cpu_time_us":5611,"apply.run_wall_time_us":414155,"num_ops":7,"prepare.queue_time_us":897899,"prepare.run_cpu_time_us":34458,"prepare.run_wall_time_us":336196,"raft.queue_time_us":570216,"raft.run_cpu_time_us":34444,"raft.run_wall_time_us":217003,"replication_time_us":980435,"thread_start_us":889845,"threads_started":4,"wal-append.queue_time_us":228694,"wal-append.run_cpu_time_us":30357,"wal-append.run_wall_time_us":2235623}]]}
I20260504 20:01:47.204104 15672 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.291219 15711 raft_consensus.cc:697] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:47.328322 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:47.325258 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.297742 15749 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:47.299772 15623 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:47.304768 15743 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:47.314404 15722 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:47.341928 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:47.343369 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:47.347628 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:47.349359 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:47.289953 15687 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260504 20:01:47.355923 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.363011 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:47.367972 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:47.368706 12819 raft_consensus.cc:606] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:47.368534 12865 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:47.369159 12819 raft_consensus.cc:614] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:47.370160 12741 raft_consensus.cc:1275] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:47.327337 15730 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:47.332494 15665 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:47.333578 15749 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:47.351822 15627 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:47.428347 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:47.326972 15563 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.028s
I20260504 20:01:47.336231 15623 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:47.339432 15682 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:47.385541 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:47.375384 15700 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:47.432066 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 74496.1us
I20260504 20:01:47.355376 15755 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:47.361758 15543 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 20:01:47.437708 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 56845us
W20260504 20:01:47.443426 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 75026.5us
I20260504 20:01:47.450502 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:47.354521 15722 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.386529 15749 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:47.352492 15743 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.484685 12863 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:47.369966 15696 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260504 20:01:47.490371 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:47.492936 12819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:01:47.493785 12819 raft_consensus.cc:606] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
W20260504 20:01:47.486809 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 44722.8us
I20260504 20:01:47.447320 15755 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:47.504223 12863 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:47.426993 15699 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260504 20:01:47.426803 15483 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
W20260504 20:01:47.507121 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 56776.8us
I20260504 20:01:47.455857 15009 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 20:01:47.516862 12865 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:47.387584 15730 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:47.532681 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:47.437949 15627 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.505003 12648 catalog_manager.cc:5671] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 3 to 4, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:47.480628 15736 consensus_queue.cc:1059] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260504 20:01:47.476392 15668 consensus_queue.cc:1048] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260504 20:01:47.436525 15665 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.457222 15700 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.513908 15025 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260504 20:01:47.516844 15755 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:47.571305 12895 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 1.
I20260504 20:01:47.572414 12648 catalog_manager.cc:5671] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
W20260504 20:01:47.559780 15771 log.cc:927] Time spent T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260504 20:01:47.449329 15682 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:47.488700 15698 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.069s
I20260504 20:01:47.553790 15681 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:47.585695 12648 catalog_manager.cc:5671] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 3 to 4, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:47.563802 15654 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:47.575501 15693 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:47.602548 12741 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
mode: GRACEFUL
new_leader_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:45440
I20260504 20:01:47.603382 12741 raft_consensus.cc:606] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Received request to transfer leadership to TS 732c3560a688494196d22a1a48f89968
I20260504 20:01:47.603794 12741 raft_consensus.cc:614] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:01:47.556277 15386 consensus_queue.cc:1048] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260504 20:01:47.484215 15647 consensus_queue.cc:260] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:47.605075 12648 catalog_manager.cc:5671] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 2 to 3, leader changed from ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:47.618497 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:47.483235 15711 consensus_queue.cc:237] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:47.631706 15775 log.cc:927] Time spent T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.252s user 0.001s sys 0.000s
W20260504 20:01:47.632295 15773 log.cc:927] Time spent T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260504 20:01:47.632706 15776 log.cc:927] Time spent T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260504 20:01:47.609213 15672 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.603569 15535 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:47.616075 15386 consensus_queue.cc:1243] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer 5ff0ef72ec3b4667893a0420d3c7f923 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260504 20:01:47.610121 15654 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:47.641108 12648 catalog_manager.cc:5671] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:47.646296 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
W20260504 20:01:47.633751 15778 log.cc:927] Time spent T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260504 20:01:47.661650 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:48.101754 12819 raft_consensus.cc:2410] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260504 20:01:47.733754 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 274502us
W20260504 20:01:47.927469 12890 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.006s sys 0.010s
W20260504 20:01:48.108705 12890 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.821s user 0.007s sys 0.012s
W20260504 20:01:47.891289 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 374473us
I20260504 20:01:48.111830 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:01:48.145313 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260504 20:01:47.712628 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 204866us
W20260504 20:01:47.929659 12818 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.007s sys 0.010s
W20260504 20:01:48.147696 12818 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.828s user 0.011s sys 0.012s
W20260504 20:01:48.186055 12816 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.006s sys 0.016s
I20260504 20:01:48.295239 12648 catalog_manager.cc:5671] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 3 to 4, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
W20260504 20:01:48.549168 12816 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.989s user 0.011s sys 0.020s
I20260504 20:01:48.550382 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 2053) took 1295 ms. Trace:
I20260504 20:01:48.549489 12791 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:48.550959 12816 rpcz_store.cc:276] 0504 20:01:47.255034 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.256052 (+ 1018us) service_pool.cc:224] Handling call
0504 20:01:47.256440 (+ 388us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:47.256727 (+ 287us) raft_consensus.cc:1534] Early marking committed up to index 9
0504 20:01:47.559742 (+303015us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.549868 (+990126us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:48.549949 (+ 81us) raft_consensus.cc:1645] Marking committed up to 9
0504 20:01:48.550019 (+ 70us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.550118 (+ 99us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.550292 (+ 174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9111,"prepare.queue_time_us":303670,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":69431,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"thread_start_us":1255493,"threads_started":2}
W20260504 20:01:48.245637 12745 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.002s sys 0.025s
W20260504 20:01:48.556404 12745 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.945s user 0.002s sys 0.034s
W20260504 20:01:48.186575 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 479511us
W20260504 20:01:48.544929 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:48.599828 12648 catalog_manager.cc:5671] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 3 to 4, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
W20260504 20:01:48.463181 15623 consensus_meta.cc:287] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.008s user 0.009s sys 0.010s
W20260504 20:01:48.342545 15789 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.506s user 0.000s sys 0.001s
I20260504 20:01:48.608493 12648 catalog_manager.cc:5671] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
W20260504 20:01:48.310307 15542 consensus_meta.cc:287] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.214s user 0.007s sys 0.008s
I20260504 20:01:47.636792 15722 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:47.728349 15635 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:48.538763 15710 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:48.619896 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1998) took 1379 ms. Trace:
I20260504 20:01:48.624307 12817 rpcz_store.cc:276] 0504 20:01:47.240209 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.249423 (+ 9214us) service_pool.cc:224] Handling call
0504 20:01:47.249982 (+ 559us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.250370 (+ 388us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.250397 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:47.250851 (+ 454us) log.cc:844] Serialized 26 byte log entry
0504 20:01:47.490481 (+239630us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:47.490522 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:47.490604 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:48.619655 (+1129051us) raft_consensus.cc:1685] finished
0504 20:01:48.619676 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.619829 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7713,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":11845,"thread_start_us":186650,"threads_started":1,"wal-append.queue_time_us":504447}
W20260504 20:01:48.278313 12748 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.018s sys 0.025s
W20260504 20:01:48.662446 12748 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.051s user 0.026s sys 0.028s
I20260504 20:01:48.247232 15672 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:48.365476 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1989) took 1128 ms. Trace:
I20260504 20:01:48.539860 15797 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:48.595553 15700 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:48.277982 15778 log.cc:927] Time spent T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.161s user 0.002s sys 0.000s
W20260504 20:01:47.848304 15788 log.cc:927] Time spent T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20260504 20:01:48.641816 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 92435.7us
I20260504 20:01:48.700045 12791 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:48.385610 15523 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.000s sys 0.010s
W20260504 20:01:48.392360 15755 consensus_meta.cc:287] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.793s user 0.010s sys 0.002s
W20260504 20:01:48.390770 15782 log.cc:927] Time spent T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260504 20:01:48.364243 15717 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:48.614655 15787 log.cc:927] Time spent T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.682s user 0.005s sys 0.001s
I20260504 20:01:48.458284 15795 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:48.107151 15647 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:48.656810 12648 catalog_manager.cc:5671] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 5 to 6, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 6 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:48.108402 15560 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:48.711706 15560 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:48.678634 12818 rpcz_store.cc:276] 0504 20:01:47.237565 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.275312 (+ 37747us) service_pool.cc:224] Handling call
0504 20:01:47.275632 (+ 320us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:47.275858 (+ 226us) raft_consensus.cc:1534] Early marking committed up to index 2
0504 20:01:48.148309 (+872451us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.364636 (+216327us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:48.364764 (+ 128us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:48.364896 (+ 132us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.365041 (+ 145us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.365353 (+ 312us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18104,"prepare.queue_time_us":327506,"raft.queue_time_us":88023,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":87162,"spinlock_wait_cycles":33796352,"thread_start_us":1044130,"threads_started":2}
I20260504 20:01:48.714915 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 320) took 1433 ms. Trace:
I20260504 20:01:48.184962 15535 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:48.649706 15722 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260504 20:01:48.694911 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 575034us
I20260504 20:01:48.726564 12890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 499) took 1476 ms. Trace:
W20260504 20:01:48.703475 15523 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.849s user 0.000s sys 0.017s
I20260504 20:01:48.727025 12890 rpcz_store.cc:276] 0504 20:01:47.250199 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.250331 (+ 132us) service_pool.cc:224] Handling call
0504 20:01:47.250608 (+ 277us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:47.250782 (+ 174us) raft_consensus.cc:1534] Early marking committed up to index 6
0504 20:01:48.186461 (+935679us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.726191 (+539730us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:48.726247 (+ 56us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:48.726293 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.726372 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.726500 (+ 128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":539433,"prepare.queue_time_us":1063655,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"thread_start_us":826594,"threads_started":1,"wal-append.queue_time_us":60688}
I20260504 20:01:48.607551 15736 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.070s
I20260504 20:01:48.342546 15696 raft_consensus.cc:993] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923: : Instructing follower 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:48.706955 12791 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:48.708925 12894 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.881s user 0.024s sys 0.044s
W20260504 20:01:48.733649 12894 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.906s user 0.025s sys 0.044s
I20260504 20:01:48.718470 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1738) took 1438 ms. Trace:
I20260504 20:01:48.715448 12821 rpcz_store.cc:276] 0504 20:01:47.281610 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.282079 (+ 469us) service_pool.cc:224] Handling call
0504 20:01:47.292420 (+ 10341us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.295173 (+ 2753us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.295212 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:47.295791 (+ 579us) log.cc:844] Serialized 26 byte log entry
0504 20:01:47.547802 (+252011us) raft_consensus.cc:1645] Marking committed up to 8
0504 20:01:47.855872 (+308070us) log.cc:844] Serialized 14 byte log entry
0504 20:01:47.963638 (+107766us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:47.963729 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:48.714613 (+750884us) raft_consensus.cc:1685] finished
0504 20:01:48.714645 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.714826 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30224,"prepare.queue_time_us":762269,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":456925,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":10489,"thread_start_us":449160,"threads_started":2,"wal-append.queue_time_us":348427}
I20260504 20:01:48.729018 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:48.738907 12890 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:48.618950 15623 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:48.652680 15635 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:48.734640 12791 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:48.671380 15710 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:48.737026 12745 rpcz_store.cc:276] 0504 20:01:47.280462 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.292345 (+ 11883us) service_pool.cc:224] Handling call
0504 20:01:47.307963 (+ 15618us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:47.318475 (+ 10512us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.318512 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:47.319132 (+ 620us) log.cc:844] Serialized 52 byte log entry
0504 20:01:47.566941 (+247809us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:01:48.599413 (+1032472us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.662421 (+ 63008us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.662538 (+ 117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:48.718147 (+ 55609us) raft_consensus.cc:1685] finished
0504 20:01:48.718178 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.718386 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14487,"raft.queue_time_us":5286,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7333,"thread_start_us":1220892,"threads_started":3,"wal-append.queue_time_us":329198}
I20260504 20:01:48.688756 15672 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:48.731258 15411 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:48.758092 12648 catalog_manager.cc:5671] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:48.720778 15717 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:48.713836 15755 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:48.807941 15623 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:48.808202 15755 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:48.745934 15009 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:48.811147 15717 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:48.812264 15535 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:48.813040 15654 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:48.802747 15782 log.cc:927] Time spent T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20260504 20:01:48.813652 15672 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:48.754297 15801 consensus_queue.cc:1048] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.061s
W20260504 20:01:48.817325 15749 consensus_meta.cc:287] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.315s user 0.012s sys 0.004s
I20260504 20:01:48.818193 12892 raft_consensus.cc:2368] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6.
I20260504 20:01:48.819353 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 501) took 1565 ms. Trace:
I20260504 20:01:48.818151 15749 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:48.821302 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1979) took 1478 ms. Trace:
W20260504 20:01:48.821537 15813 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.150s user 0.000s sys 0.001s
I20260504 20:01:48.821794 12895 rpcz_store.cc:276] 0504 20:01:47.342824 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.343014 (+ 190us) service_pool.cc:224] Handling call
0504 20:01:48.821064 (+1478050us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b540013bad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:48.821214 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522565120}
I20260504 20:01:48.823318 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:48.823426 15749 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:48.824035 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1977) took 1481 ms. Trace:
I20260504 20:01:48.824254 12895 raft_consensus.cc:2410] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260504 20:01:48.824460 15682 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:48.824694 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1980) took 1481 ms. Trace:
I20260504 20:01:48.698021 15797 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:48.819774 12898 rpcz_store.cc:276] 0504 20:01:47.254173 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.254499 (+ 326us) service_pool.cc:224] Handling call
0504 20:01:47.254955 (+ 456us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.262294 (+ 7339us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.262329 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:47.262851 (+ 522us) log.cc:844] Serialized 26 byte log entry
0504 20:01:47.508171 (+245320us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:47.869123 (+360952us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.545333 (+676210us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.545452 (+ 119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:48.819100 (+273648us) raft_consensus.cc:1685] finished
0504 20:01:48.819126 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.819280 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71998,"prepare.queue_time_us":807053,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":790684,"raft.run_cpu_time_us":433,"raft.run_wall_time_us":43223,"thread_start_us":550420,"threads_started":2,"wal-append.queue_time_us":922294}
I20260504 20:01:48.838702 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:48.826743 12648 catalog_manager.cc:5671] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } }
I20260504 20:01:48.824434 12896 rpcz_store.cc:276] 0504 20:01:47.342202 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.342357 (+ 155us) service_pool.cc:224] Handling call
0504 20:01:47.342667 (+ 310us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.365830 (+ 23163us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.365880 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:47.366484 (+ 604us) log.cc:844] Serialized 26 byte log entry
0504 20:01:47.710857 (+344373us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b1800255160. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d471 00007f015c969def
0504 20:01:48.025495 (+314638us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:48.641468 (+615973us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.735876 (+ 94408us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.735972 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:48.823709 (+ 87737us) raft_consensus.cc:1685] finished
0504 20:01:48.823845 (+ 136us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.823975 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":329287,"prepare.queue_time_us":432812,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":226642,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":24130,"spinlock_wait_cycles":8855936,"thread_start_us":644265,"threads_started":2,"wal-append.queue_time_us":1053739}
I20260504 20:01:48.840562 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:48.840776 12898 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:48.825338 12895 rpcz_store.cc:276] 0504 20:01:47.343066 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.822990 (+1479924us) service_pool.cc:224] Handling call
0504 20:01:48.824641 (+ 1651us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:48.841310 12896 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:48.842092 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1983) took 1498 ms. Trace:
I20260504 20:01:48.842852 12898 rpcz_store.cc:276] 0504 20:01:47.343918 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.837472 (+1493554us) service_pool.cc:224] Handling call
0504 20:01:48.841961 (+ 4489us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:48.782704 15722 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260504 20:01:48.858877 13497 connection.cc:254] RPC call timeout handler was delayed by -1.226842357999999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
I20260504 20:01:48.861270 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:48.863551 12895 raft_consensus.cc:2410] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260504 20:01:48.864639 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1987) took 1519 ms. Trace:
I20260504 20:01:48.865257 12895 rpcz_store.cc:276] 0504 20:01:47.345123 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.842384 (+1497261us) service_pool.cc:224] Handling call
0504 20:01:48.864542 (+ 22158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:48.869107 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 646) took 1786 ms. Trace:
I20260504 20:01:48.869577 13530 rpcz_store.cc:276] 0504 20:01:47.083065 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.083760 (+ 695us) service_pool.cc:224] Handling call
0504 20:01:47.084127 (+ 367us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.084521 (+ 394us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.084556 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:47.099001 (+ 14445us) log.cc:844] Serialized 26 byte log entry
0504 20:01:47.378124 (+279123us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:47.487559 (+109435us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.857963 (+1370404us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b18003b06e0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159884e1a 00007f0162f265c9 00007f01597431f0 00007f015990db24 00007f015990da95 00007f015c9a6440 00007f015c9b8af6 00007f015c9c9667 00007f015c9c95e8
0504 20:01:48.858179 (+ 216us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.858251 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:48.858294 (+ 43us) raft_consensus.cc:1685] finished
0504 20:01:48.868829 (+ 10535us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.869013 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4355,"prepare.queue_time_us":60378,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":1367855,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":100,"spinlock_wait_cycles":12067200,"thread_start_us":240089,"threads_started":1,"wal-append.queue_time_us":376194}
W20260504 20:01:48.875372 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 1.27088e+06us
I20260504 20:01:48.752422 15386 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.102s
W20260504 20:01:48.860612 15789 log.cc:927] Time spent T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260504 20:01:48.890034 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1986) took 1545 ms. Trace:
I20260504 20:01:48.731374 15795 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:48.894512 12899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1976) took 1552 ms. Trace:
I20260504 20:01:48.913558 12890 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
W20260504 20:01:48.920289 12890 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1984) took 1576 ms (client timeout 1868 ms). Trace:
W20260504 20:01:48.920750 12890 rpcz_store.cc:269] 0504 20:01:47.344207 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.728733 (+1384526us) service_pool.cc:224] Handling call
0504 20:01:48.920217 (+191484us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:48.919528 12899 rpcz_store.cc:276] 0504 20:01:47.341861 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.346134 (+ 4273us) service_pool.cc:224] Handling call
0504 20:01:47.363766 (+ 17632us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:47.371962 (+ 8196us) raft_consensus.cc:1534] Early marking committed up to index 3
0504 20:01:47.743999 (+372037us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.894139 (+1150140us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:48.894190 (+ 51us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:48.894239 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.894313 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.894447 (+ 134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":519117,"prepare.queue_time_us":942762,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":523816,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":10420,"thread_start_us":945119,"threads_started":3,"wal-append.queue_time_us":1128847}
W20260504 20:01:48.927572 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 199703us
I20260504 20:01:48.923563 12818 raft_consensus.cc:3055] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:48.928143 12818 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:48.928428 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:48.924185 13497 connection.cc:254] RPC call timeout handler was delayed by -1.147083160999955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
I20260504 20:01:48.925647 12890 raft_consensus.cc:1240] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:48.929566 12890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1749) took 1583 ms. Trace:
I20260504 20:01:48.930156 12890 rpcz_store.cc:276] 0504 20:01:47.345763 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.924469 (+1578706us) service_pool.cc:224] Handling call
0504 20:01:48.925246 (+ 777us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:48.929303 (+ 4057us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.929507 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:48.931128 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:48.931521 12890 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260504 20:01:48.931854 12890 pending_rounds.cc:85] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 5
I20260504 20:01:48.932065 12890 pending_rounds.cc:107] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260504 20:01:48.932291 12890 raft_consensus.cc:2889] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:48.914183 12895 rpcz_store.cc:276] 0504 20:01:47.344814 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.866977 (+1522163us) service_pool.cc:224] Handling call
0504 20:01:48.867712 (+ 735us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:48.868177 (+ 465us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:48.868286 (+ 109us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:48.868412 (+ 126us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:48.868500 (+ 88us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.889258 (+ 20758us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:48.889939 (+ 681us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:48.916612 12648 catalog_manager.cc:5671] T 8fd38813d75c466e8f0502c36726a913 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 1 to 2, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 2 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:48.939857 15672 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:48.940589 15672 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:48.939431 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:48.944296 12865 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:48.945003 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:48.945772 12816 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:48.945596 12648 catalog_manager.cc:5671] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 4 to 6, leader changed from 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 6 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:48.909065 15179 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:48.949043 12897 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:48.950816 12817 raft_consensus.cc:2410] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260504 20:01:48.919209 15560 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:48.960757 12648 catalog_manager.cc:5671] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 reported cstate change: term changed from 2 to 4, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 4 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } }
I20260504 20:01:48.965232 12823 pending_rounds.cc:85] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 1
I20260504 20:01:48.965649 12823 pending_rounds.cc:107] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260504 20:01:48.965973 12823 raft_consensus.cc:2889] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:48.966678 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:48.958042 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:49.007625 15536 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:49.009016 15814 log.cc:927] Time spent T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20260504 20:01:49.012070 15816 log.cc:927] Time spent T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.275s user 0.001s sys 0.000s
I20260504 20:01:49.010516 15801 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260504 20:01:49.097581 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 364268us
I20260504 20:01:49.054953 12899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 502) took 1470 ms. Trace:
W20260504 20:01:49.054204 13497 leader_election.cc:336] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.574s (SENT)
I20260504 20:01:48.964387 15439 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:48.959930 15749 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:48.942169 12818 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.078796 12822 raft_consensus.cc:3055] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:49.101110 12822 raft_consensus.cc:740] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:49.077744 12816 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:48.963985 15605 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:49.090417 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 135856us
I20260504 20:01:49.090345 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1732) took 1811 ms. Trace:
I20260504 20:01:49.105062 12749 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:01:49.105175 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.106530 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.107522 12749 raft_consensus.cc:2410] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260504 20:01:49.105942 12743 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 1.
I20260504 20:01:49.053138 15826 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:49.063992 15542 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:49.155035 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:49.152829 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:49.155885 12743 raft_consensus.cc:2393] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:49.074911 15810 log.cc:927] Time spent T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.361s user 0.000s sys 0.003s
I20260504 20:01:49.098997 13497 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:49.109659 12747 raft_consensus.cc:1275] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:01:49.070960 15812 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:49.106818 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.166990 12746 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
W20260504 20:01:49.167150 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 112989us
I20260504 20:01:49.168224 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:49.168998 12746 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:49.072188 12714 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.028s sys 0.048s
W20260504 20:01:49.171854 12714 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s user 0.029s sys 0.048s
I20260504 20:01:49.053620 15800 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:49.087396 15819 log.cc:927] Time spent T 445dd93bcaf04bae88b84d4eb9eddb60 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.291s user 0.002s sys 0.001s
I20260504 20:01:49.147133 12818 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:49.146910 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.147711 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.100162 12899 rpcz_store.cc:276] 0504 20:01:47.584350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.926536 (+1342186us) service_pool.cc:224] Handling call
0504 20:01:49.054332 (+127796us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:49.054567 (+ 235us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:49.054608 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:49.054664 (+ 56us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:49.054706 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:49.054761 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.054893 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:49.095818 12825 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:49.052878 15380 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:49.153944 12822 consensus_queue.cc:260] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.080811 15710 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.089747 15700 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:49.144455 15793 raft_consensus.cc:1081] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Signalling peer 732c3560a688494196d22a1a48f89968 to start an election
I20260504 20:01:49.228184 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:49.228966 12749 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:49.174414 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:49.162616 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.108799 15560 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:49.229952 12747 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:49.229846 12743 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:49.231014 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1973) took 1890 ms. Trace:
I20260504 20:01:49.232213 12822 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:49.108420 15717 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:49.244244 12818 raft_consensus.cc:1275] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260504 20:01:49.245157 12818 pending_rounds.cc:85] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 3
I20260504 20:01:49.245698 12818 pending_rounds.cc:107] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260504 20:01:49.246057 12818 raft_consensus.cc:2889] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:49.242949 12893 rpcz_store.cc:276] 0504 20:01:47.340773 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.345925 (+ 5152us) service_pool.cc:224] Handling call
0504 20:01:47.362666 (+ 16741us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.457717 (+ 95051us) raft_consensus.cc:1534] Early marking committed up to index 3
0504 20:01:48.398825 (+941108us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b18002d8da0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f015990da95 00007f01603b1837 00007f015c9b8d87 00007f015c9c849f 00007f015c9c8418
0504 20:01:48.457326 (+ 58501us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.875461 (+418135us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:48.876093 (+ 632us) log.cc:844] Serialized 26 byte log entry
0504 20:01:48.876330 (+ 237us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:48.885296 (+ 8966us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.909614 (+ 24318us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.909700 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:49.230701 (+321001us) raft_consensus.cc:1685] finished
0504 20:01:49.230737 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.230933 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":286375,"prepare.queue_time_us":862103,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":258447,"raft.run_cpu_time_us":553,"raft.run_wall_time_us":30866,"spinlock_wait_cycles":4274944,"thread_start_us":390859,"threads_started":3,"wal-append.queue_time_us":330295}
I20260504 20:01:49.164484 15696 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:49.250449 12746 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:49.150300 15682 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:49.164922 15687 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:49.268469 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 307582us
I20260504 20:01:49.262028 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:49.096724 15536 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.145087 15439 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:49.240072 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:49.296556 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 204111us
I20260504 20:01:49.316432 12822 raft_consensus.cc:1275] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
W20260504 20:01:49.239647 15787 log.cc:927] Time spent T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260504 20:01:49.342974 12822 pending_rounds.cc:85] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 3
I20260504 20:01:49.376356 12822 pending_rounds.cc:107] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260504 20:01:49.376772 12822 raft_consensus.cc:2889] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:49.240669 15635 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.342154 12743 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:49.164521 15542 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.194689 15812 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.262109 15831 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:49.402643 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:49.403517 12890 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 2.
I20260504 20:01:49.384279 12749 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:49.252301 15654 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:49.264546 15696 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:49.263592 15605 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:49.427443 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:49.428689 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:49.398679 12894 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1978) took 2056 ms. Trace:
I20260504 20:01:49.382241 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:49.403319 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 304401us
I20260504 20:01:49.279279 15684 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:49.402731 15822 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20260504 20:01:49.269325 15717 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.275668 15535 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:49.239903 15380 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.420583 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:49.406563 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:49.438220 12899 raft_consensus.cc:2410] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:49.163014 15826 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.429988 15696 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:49.446234 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:49.434185 12894 rpcz_store.cc:276] 0504 20:01:47.342541 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.342971 (+ 430us) service_pool.cc:224] Handling call
0504 20:01:47.343280 (+ 309us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.357533 (+ 14253us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.357567 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:47.358153 (+ 586us) log.cc:844] Serialized 26 byte log entry
0504 20:01:47.760811 (+402658us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:48.734120 (+973309us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.743055 (+ 8935us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.743154 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:48.819902 (+ 76748us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x7b4800210258. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f01618e61a1 00007f01618dd237 00007f01618dd0ca 00007f01619d492f 00007f01619d45f7 00007f01619e8c1c 00007f01619e8bc8 00007f01619e8b48 00007f01619e8b03 00007f01619e7dac 00007f0160910b36
0504 20:01:49.389194 (+569292us) raft_consensus.cc:1685] finished
0504 20:01:49.398374 (+ 9180us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.398586 (+ 212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22505,"prepare.queue_time_us":554658,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":769,"raft.queue_time_us":113952,"raft.run_cpu_time_us":20676,"raft.run_wall_time_us":1437668,"spinlock_wait_cycles":125562624,"thread_start_us":1282235,"threads_started":2,"wal-append.queue_time_us":1241189}
I20260504 20:01:49.416917 15833 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:49.282668 15797 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.403841 15710 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:49.248821 15700 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.226131 15800 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.458444 15672 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.334643 15682 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.393777 15832 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:49.403808 15635 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:49.463620 12890 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
W20260504 20:01:49.441856 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 65930us
I20260504 20:01:49.469626 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:49.431078 15824 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:49.333372 15687 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.494628 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:49.264278 15560 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.463397 15523 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:49.444686 15654 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.448357 15605 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:49.529995 12896 consensus_meta.cc:287] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.662s user 0.010s sys 0.003s
I20260504 20:01:49.493757 15795 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.452459 15684 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.537685 12896 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:49.468266 15535 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:01:49.424383 15831 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.538182 15824 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.544411 15825 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:49.562372 15560 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.562036 15825 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.525110 15832 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.564980 15682 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.565975 15825 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.545598 15838 consensus_queue.cc:1059] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260504 20:01:49.546383 15717 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:49.546849 15815 log.cc:927] Time spent T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.467s user 0.000s sys 0.003s
I20260504 20:01:49.528091 15635 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.569482 15439 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:49.570197 15635 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.526248 15672 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:49.571367 15672 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.574906 15542 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:49.562165 15536 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.575564 15672 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.548182 15835 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:49.549425 13532 raft_consensus.cc:1813] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Handling vote request from an unknown peer 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:49.577351 13532 raft_consensus.cc:2410] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260504 20:01:49.577040 15835 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.578814 15832 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.580971 15835 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:49.551070 12896 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 1982) took 2207 ms (client timeout 2534 ms). Trace:
I20260504 20:01:49.503422 15710 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.554486 15687 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.548931 15836 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260504 20:01:49.537899 15647 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.815s user 0.006s sys 0.011s
W20260504 20:01:49.590756 12896 rpcz_store.cc:269] 0504 20:01:47.343618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.840206 (+1496588us) service_pool.cc:224] Handling call
0504 20:01:49.550995 (+710789us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:49.605157 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1981) took 2261 ms. Trace:
I20260504 20:01:49.614133 12898 rpcz_store.cc:276] 0504 20:01:47.343321 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.844136 (+1500815us) service_pool.cc:224] Handling call
0504 20:01:48.844535 (+ 399us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:48.869550 (+ 25015us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:48.869579 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:48.872860 (+ 3281us) log.cc:844] Serialized 26 byte log entry
0504 20:01:49.400252 (+527392us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:49.419476 (+ 19224us) log.cc:844] Serialized 14 byte log entry
0504 20:01:49.604726 (+185250us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:49.604839 (+ 113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:49.604904 (+ 65us) raft_consensus.cc:1685] finished
0504 20:01:49.604920 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.605075 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69001,"prepare.queue_time_us":18940,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":166150,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":48904,"thread_start_us":621058,"threads_started":2,"wal-append.queue_time_us":678766}
I20260504 20:01:49.492273 15833 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.577693 15824 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.614840 15830 raft_consensus.cc:3055] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:49.542307 15523 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.604118 15621 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20260504 20:01:49.643246 15830 raft_consensus.cc:740] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:49.663141 15812 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.651544 15811 consensus_queue.cc:1048] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.190s
I20260504 20:01:49.675603 15830 consensus_queue.cc:260] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.676786 15830 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:49.678936 15831 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:49.660383 15841 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.180s
I20260504 20:01:49.682035 15826 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.683815 15833 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.467940 12716 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:49.719062 12715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.022s sys 0.022s
W20260504 20:01:49.720116 12715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s user 0.022s sys 0.022s
W20260504 20:01:49.722101 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 255086us
I20260504 20:01:49.724298 12716 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:49.727897 12716 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:49.731458 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 445) took 2120 ms. Trace:
I20260504 20:01:49.732179 12742 rpcz_store.cc:276] 0504 20:01:47.610282 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.610403 (+ 121us) service_pool.cc:224] Handling call
0504 20:01:47.646674 (+ 36271us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.647136 (+ 462us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.647164 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:48.099762 (+452598us) log.cc:844] Serialized 26 byte log entry
0504 20:01:48.353940 (+254178us) raft_consensus.cc:1645] Marking committed up to 8
0504 20:01:48.354002 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.354122 (+ 120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:49.730617 (+1376495us) raft_consensus.cc:1685] finished
0504 20:01:49.730694 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.731122 (+ 428us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36394,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":17310,"thread_start_us":253637,"threads_started":1,"wal-append.queue_time_us":791722}
I20260504 20:01:49.484293 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.735742 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.481971 12743 raft_consensus.cc:1240] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260504 20:01:49.737500 12747 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:49.739562 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.740554 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.741156 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.742236 12747 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:49.742413 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.743831 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.744031 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.745406 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.480638 12749 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:49.745759 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.750140 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.752110 12750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1739) took 2404 ms. Trace:
I20260504 20:01:49.761617 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:49.763284 12749 raft_consensus.cc:2393] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:49.760330 12750 rpcz_store.cc:276] 0504 20:01:47.347293 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.347966 (+ 673us) service_pool.cc:224] Handling call
0504 20:01:47.372455 (+ 24489us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:47.372859 (+ 404us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.372904 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:47.381883 (+ 8979us) log.cc:844] Serialized 52 byte log entry
0504 20:01:47.730390 (+348507us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:01:49.467513 (+1737123us) log.cc:844] Serialized 14 byte log entry
0504 20:01:49.546282 (+ 78769us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b180046e940. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159910a33 00007f0159912e01 00007f0159912db9 00007f0159912d49 00007f0159912d11 00007f015991200d 0000000000379254 0000000000379099 00007f01598f3fba
0504 20:01:49.750832 (+204550us) log.cc:844] Serialized 14 byte log entry
0504 20:01:49.751245 (+ 413us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:49.751417 (+ 172us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:49.751623 (+ 206us) raft_consensus.cc:1685] finished
0504 20:01:49.751645 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.751898 (+ 253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1417243,"prepare.queue_time_us":1610131,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":92,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":52811,"spinlock_wait_cycles":87149824,"thread_start_us":789546,"threads_started":3,"wal-append.queue_time_us":1068770,"wal-append.run_cpu_time_us":2059,"wal-append.run_wall_time_us":1025025}
I20260504 20:01:49.764657 12742 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:49.766137 12745 raft_consensus.cc:1240] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:49.767529 12743 raft_consensus.cc:2410] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:49.594044 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:49.784797 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 190648us
I20260504 20:01:49.786711 15831 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:49.787729 15831 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:49.788228 15654 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:49.788522 15831 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:49.790639 15672 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:49.791992 15672 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 20:01:49.792528 15672 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:49.793043 15672 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 1 FOLLOWER]: Advancing to term 2
W20260504 20:01:49.798157 13499 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.002s sys 0.003s
I20260504 20:01:49.798945 15700 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:49.802381 13497 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.005s sys 0.005s
W20260504 20:01:49.803056 13497 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.635s user 0.005s sys 0.005s
W20260504 20:01:49.803661 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 635641us
I20260504 20:01:49.813596 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:49.798952 13499 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.002s sys 0.003s
W20260504 20:01:49.842962 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 115378us
I20260504 20:01:49.845286 15380 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.845695 15710 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.848097 15684 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.852236 15523 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:49.854179 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 76262.3us
I20260504 20:01:49.855011 15605 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.855926 15834 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:49.858462 12892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1748) took 2512 ms (client timeout 1676 ms). Trace:
I20260504 20:01:49.861577 15635 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:49.861936 15535 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:49.825502 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:49.872112 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 594019us
W20260504 20:01:49.868805 12892 rpcz_store.cc:269] 0504 20:01:47.345472 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.474856 (+129384us) service_pool.cc:224] Handling call
0504 20:01:49.857564 (+2382708us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540013c9d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:49.858026 (+ 462us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3051527424}
W20260504 20:01:49.875428 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:49.876507 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:49.877059 15833 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:49.875167 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:49.877895 12791 connection.cc:254] RPC call timeout handler was delayed by -1.8401286089999758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
I20260504 20:01:49.881212 13532 raft_consensus.cc:2410] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260504 20:01:49.878258 15830 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:49.884876 15830 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:49.886564 13497 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.975s (SENT) [suppressed 7 similar messages]
I20260504 20:01:49.888453 15830 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.885450 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:49.893355 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:49.894922 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:49.895545 15736 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:49.896956 15699 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:49.898104 15699 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:49.899966 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 314) took 2822 ms. Trace:
I20260504 20:01:49.889750 15627 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:49.909643 15833 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.911424 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260504 20:01:49.911618 15777 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 2.363s user 0.004s sys 0.014s
I20260504 20:01:49.879345 15834 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:49.911285 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:49.913744 15833 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.914753 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:49.916432 15834 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:49.927114 15726 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:49.928009 15803 log.cc:927] Time spent T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 1.584s user 0.000s sys 0.017s
I20260504 20:01:49.900537 12820 rpcz_store.cc:276] 0504 20:01:47.077769 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.103394 (+ 25625us) service_pool.cc:224] Handling call
0504 20:01:47.122707 (+ 19313us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.123085 (+ 378us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.123115 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:47.123617 (+ 502us) log.cc:844] Serialized 26 byte log entry
0504 20:01:47.389295 (+265678us) raft_consensus.cc:1645] Marking committed up to 8
0504 20:01:47.592907 (+203612us) log.cc:844] Serialized 14 byte log entry
0504 20:01:49.851693 (+2258786us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b18003b2f00. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f015990da95 00007f015c9a6440 00007f015c9b8af6 00007f015c9c9667 00007f015c9c95e8
0504 20:01:49.894064 (+ 42371us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:49.899142 (+ 5078us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:49.899278 (+ 136us) raft_consensus.cc:1685] finished
0504 20:01:49.899466 (+ 188us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.899699 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":920267,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":50648,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":12057,"spinlock_wait_cycles":17473024,"thread_start_us":372370,"threads_started":3,"wal-append.queue_time_us":337209,"wal-append.run_cpu_time_us":41381,"wal-append.run_wall_time_us":2353912}
I20260504 20:01:49.929785 15726 consensus_queue.cc:1048] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260504 20:01:49.931214 12865 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:49.930421 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 1972) took 2589 ms. Trace:
I20260504 20:01:49.933000 12891 rpcz_store.cc:276] 0504 20:01:47.340386 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.340691 (+ 305us) service_pool.cc:224] Handling call
0504 20:01:47.341059 (+ 368us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:47.365036 (+ 23977us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.365088 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:47.385032 (+ 19944us) log.cc:844] Serialized 26 byte log entry
0504 20:01:47.963752 (+578720us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:48.459428 (+495676us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.731458 (+272030us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:48.731562 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:49.926828 (+1195266us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:49.929649 (+ 2821us) raft_consensus.cc:1685] finished
0504 20:01:49.929786 (+ 137us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.930136 (+ 350us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":318078,"prepare.queue_time_us":421462,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":6395,"raft.queue_time_us":339025,"raft.run_cpu_time_us":730,"raft.run_wall_time_us":92862,"spinlock_wait_cycles":42609920,"thread_start_us":622142,"threads_started":2,"wal-append.queue_time_us":888376}
I20260504 20:01:49.932564 15605 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:49.936198 15605 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:49.936656 15605 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:49.940260 15665 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:49.942399 12862 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.006s sys 0.014s
W20260504 20:01:49.942940 12862 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.006s sys 0.014s
W20260504 20:01:49.952785 12714 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.021s sys 0.010s
W20260504 20:01:49.953406 12714 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s user 0.022s sys 0.010s
I20260504 20:01:49.963774 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:49.917410 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:49.966830 15483 raft_consensus.cc:3055] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:49.967571 15483 raft_consensus.cc:740] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:49.969750 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:49.971252 13526 raft_consensus.cc:2410] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260504 20:01:49.905716 15699 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:49.982971 15833 consensus_queue.cc:1059] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:49.984563 15833 raft_consensus.cc:3055] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:01:49.985337 15833 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:49.102581 12744 rpcz_store.cc:276] 0504 20:01:47.278513 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.281697 (+ 3184us) service_pool.cc:224] Handling call
0504 20:01:47.296822 (+ 15125us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:47.297297 (+ 475us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:47.297337 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:47.319508 (+ 22171us) log.cc:844] Serialized 52 byte log entry
0504 20:01:47.563059 (+243551us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:47.591701 (+ 28642us) log.cc:844] Serialized 14 byte log entry
0504 20:01:48.541369 (+949668us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:49.089464 (+548095us) spinlock_profiling.cc:243] Waited 902 us on lock 0x7b1800342ac0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159884e1a 00007f0162f265c9 00007f01597431f0 00007f015990db24 00007f015990da95 00007f01603b1837 00007f015c9b8d87 00007f015c9c849f 00007f015c9c8418
0504 20:01:49.089701 (+ 237us) log.cc:844] Serialized 14 byte log entry
0504 20:01:49.089956 (+ 255us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:49.090036 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:49.090084 (+ 48us) raft_consensus.cc:1685] finished
0504 20:01:49.090096 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:49.090283 (+ 187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19146,"raft.queue_time_us":17619,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":24900,"spinlock_wait_cycles":705055232,"thread_start_us":208672,"threads_started":1,"wal-append.queue_time_us":599635}
I20260504 20:01:50.092904 15696 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.096053 15696 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.105923 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:50.101042 12864 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:50.098322 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:50.102785 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:49.977648 12825 consensus_meta.cc:287] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.750s user 0.005s sys 0.014s
W20260504 20:01:50.120874 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 212125us
I20260504 20:01:49.969624 15483 consensus_queue.cc:260] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:50.127631 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 210234us
I20260504 20:01:50.131568 15801 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:50.133544 13497 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:50.134034 12865 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:50.134238 15870 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:50.135216 15870 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.138238 15839 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:50.138650 15833 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.140554 15833 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:50.141660 15850 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:50.142156 15651 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:50.142557 15834 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:50.142957 15647 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:50.150135 15605 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.152782 15483 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:50.142606 15850 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.154690 12716 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:50.153973 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 67370us
I20260504 20:01:50.156440 15534 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:50.156771 12716 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:50.158090 15847 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:50.158649 15847 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:50.158748 15870 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.159188 15847 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:50.142933 15651 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.163810 15825 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:50.164579 15825 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.143381 15834 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.137387 12865 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:50.152822 15605 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:50.160630 15850 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.175026 15825 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:50.180413 15803 log.cc:927] Time spent T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260504 20:01:50.182215 15834 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.185479 15870 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:50.186420 15870 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:50.167223 15800 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:50.188813 12716 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:50.189980 15870 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:50.190882 12716 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:50.172834 15651 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.198988 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:50.200973 15672 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.201704 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:50.205642 15698 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:50.179342 15831 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.207139 15864 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:50.207931 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:50.211117 15825 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:50.262542 15825 raft_consensus.cc:697] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:50.263175 15743 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:50.263854 15825 consensus_queue.cc:237] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.264725 15831 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:50.212042 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.265782 12820 raft_consensus.cc:1813] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Handling vote request from an unknown peer 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:50.260653 15698 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:50.266444 12820 raft_consensus.cc:2410] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260504 20:01:50.213465 15668 consensus_queue.cc:1059] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:50.214295 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.217847 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.216214 12863 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:50.219672 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.218688 15800 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:50.220227 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.223452 15605 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:50.273082 15605 raft_consensus.cc:697] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:50.223901 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 86686us
I20260504 20:01:50.274392 12747 raft_consensus.cc:1275] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:50.274907 12865 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:50.227783 15696 consensus_queue.cc:1059] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260504 20:01:50.229015 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.229367 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.227872 12818 raft_consensus.cc:1240] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:50.230093 12791 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:50.230970 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.231344 15870 consensus_queue.cc:1059] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:01:50.233402 12716 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:50.251354 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:50.261142 15380 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:50.258773 13528 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:50.283428 12824 raft_consensus.cc:2410] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260504 20:01:50.210388 12825 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:50.271698 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.272102 15698 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:50.276392 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.268893 12822 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
W20260504 20:01:50.280251 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.276700 15179 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:50.296139 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.295838 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:50.297700 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 64426.8us
I20260504 20:01:50.298003 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 2006) took 1353 ms. Trace:
I20260504 20:01:50.298530 12825 rpcz_store.cc:276] 0504 20:01:48.938885 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.944112 (+ 5227us) service_pool.cc:224] Handling call
0504 20:01:50.288249 (+1344137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:50.296764 12820 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:50.295428 12819 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:50.301644 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:50.305840 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 90268.5us
I20260504 20:01:50.295746 15864 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:50.317565 12824 raft_consensus.cc:2410] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260504 20:01:50.314164 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:50.322777 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 92819.5us
I20260504 20:01:50.274407 15605 consensus_queue.cc:237] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.306959 15179 raft_consensus.cc:697] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:50.316321 15699 raft_consensus.cc:3055] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:01:50.315627 15864 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:50.331522 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.295456 15380 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:50.337661 12747 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:50.323563 15698 raft_consensus.cc:3055] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:50.332526 15668 raft_consensus.cc:3055] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:50.335326 15864 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:50.332839 15699 raft_consensus.cc:740] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:50.336158 15830 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:50.322867 15800 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.352340 15563 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260504 20:01:50.358243 15380 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.351295 15870 consensus_queue.cc:1059] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260504 20:01:50.343407 15698 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:50.346479 15668 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:50.337239 15743 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260504 20:01:50.359180 15696 raft_consensus.cc:3055] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Stepping down as leader of term 2
I20260504 20:01:50.366624 15699 consensus_queue.cc:260] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.367282 15696 raft_consensus.cc:740] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:50.368294 15699 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:50.368369 15696 consensus_queue.cc:260] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.368726 12791 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:50.349699 15830 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:50.371575 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.370024 15870 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:50.365049 15672 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.384817 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.373150 15830 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:50.391568 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:50.392362 12897 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 488) took 3323 ms (client timeout 2247 ms). Trace:
I20260504 20:01:50.382156 15684 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:50.392575 12818 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
W20260504 20:01:50.392848 12897 rpcz_store.cc:269] 0504 20:01:47.068334 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.072835 (+ 4501us) service_pool.cc:224] Handling call
0504 20:01:50.391950 (+3319115us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54001b9c54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:50.392121 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3564183424}
I20260504 20:01:50.393170 15684 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.390586 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:50.395725 15684 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:50.395982 15736 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:50.396458 15847 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.396845 15736 consensus_queue.cc:1059] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:50.399741 15386 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:50.400528 15698 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.381070 15696 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:50.375422 15870 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:50.404830 12824 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:50.414674 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.410545 15386 raft_consensus.cc:697] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:50.415532 15870 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:50.436353 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.441579 15833 raft_consensus.cc:3055] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:50.442032 12819 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:50.442271 15833 raft_consensus.cc:740] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:50.441607 15638 consensus_queue.cc:1048] T ae27dbfd0ab84c9d8c95896c0c4cf502 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260504 20:01:50.453222 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.453711 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.454120 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:50.459374 12823 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 6
I20260504 20:01:50.456139 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.457047 15833 consensus_queue.cc:260] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.451612 15647 consensus_queue.cc:1059] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
W20260504 20:01:50.474457 12716 connection.cc:254] RPC call timeout handler was delayed by -1.0217803409999533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
I20260504 20:01:50.473515 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.480760 12817 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:50.468677 15698 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:50.432006 15179 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:50.493495 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:50.489629 15800 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:50.509156 12823 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:01:50.513008 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:50.516808 15872 log.cc:927] Time spent T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20260504 20:01:50.519562 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 123179us
I20260504 20:01:50.487031 15668 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.524816 15847 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:50.539176 15785 log.cc:927] Time spent T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 2.514s user 0.000s sys 0.028s
I20260504 20:01:50.556353 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 1999) took 1642 ms. Trace:
I20260504 20:01:50.568910 12791 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:50.570917 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 294370us
I20260504 20:01:50.574155 15877 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:50.574815 15877 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:50.574875 15798 log.cc:927] Time spent T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 1.870s user 0.014s sys 0.004s
I20260504 20:01:50.574971 12824 raft_consensus.cc:1275] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260504 20:01:50.575318 15877 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:50.575553 12824 pending_rounds.cc:85] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 8
I20260504 20:01:50.575958 12824 pending_rounds.cc:107] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260504 20:01:50.576262 12824 raft_consensus.cc:2889] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:50.578017 15483 consensus_queue.cc:1059] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260504 20:01:50.579720 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.515226 15386 consensus_queue.cc:237] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.601037 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.601481 15864 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:50.602882 12743 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:50.558209 12821 rpcz_store.cc:276] 0504 20:01:48.913351 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:48.913589 (+ 238us) service_pool.cc:224] Handling call
0504 20:01:48.928814 (+ 15225us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:49.376720 (+447906us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:49.376752 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:49.403265 (+ 26513us) log.cc:844] Serialized 52 byte log entry
0504 20:01:50.270565 (+867300us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b1800344b60. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d471 00007f015c969def
0504 20:01:50.430904 (+160339us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:50.434312 (+ 3408us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:50.436699 (+ 2387us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:50.555764 (+119065us) raft_consensus.cc:1685] finished
0504 20:01:50.555795 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:50.556079 (+ 284us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11074,"raft.queue_time_us":489127,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":25184,"spinlock_wait_cycles":6260096,"thread_start_us":561201,"threads_started":2,"wal-append.queue_time_us":1070884}
I20260504 20:01:50.632210 12819 raft_consensus.cc:1275] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260504 20:01:50.633910 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:50.634825 13528 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:50.658450 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.632741 12819 pending_rounds.cc:85] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 2
I20260504 20:01:50.660050 12819 pending_rounds.cc:107] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260504 20:01:50.660414 12819 raft_consensus.cc:2889] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:50.663990 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:50.664372 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 165487us
I20260504 20:01:50.665251 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.630184 15668 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:50.673349 15850 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:50.674506 15850 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.677977 15850 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.679073 15850 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:50.679777 15850 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.683382 15850 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:50.694551 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.626005 15523 consensus_queue.cc:1059] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260504 20:01:50.707424 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.708174 12825 raft_consensus.cc:2393] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:50.710258 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.724802 12790 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:50.726166 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.726938 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:50.727352 15755 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:50.727993 15755 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:50.728704 15877 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:50.738049 15873 log.cc:927] Time spent T 00000000000000000000000000000000 P 109800bd8ea34eaea0c7308025ecde32: Append to log took a long time: real 0.315s user 0.000s sys 0.003s
I20260504 20:01:50.742570 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1723) took 3537 ms. Trace:
I20260504 20:01:50.743062 12748 rpcz_store.cc:276] 0504 20:01:47.204708 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:47.204866 (+ 158us) service_pool.cc:224] Handling call
0504 20:01:47.205217 (+ 351us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:47.205694 (+ 477us) raft_consensus.cc:1534] Early marking committed up to index 2
0504 20:01:47.225921 (+ 20227us) log.cc:844] Serialized 14 byte log entry
0504 20:01:47.569998 (+344077us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:47.578612 (+ 8614us) log.cc:844] Serialized 52 byte log entry
0504 20:01:47.578793 (+ 181us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:48.663047 (+1084254us) log.cc:844] Serialized 14 byte log entry
0504 20:01:49.264776 (+601729us) log.cc:844] Serialized 14 byte log entry
0504 20:01:49.275420 (+ 10644us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:49.275544 (+ 124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:50.360582 (+1085038us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:50.742147 (+381565us) raft_consensus.cc:1685] finished
0504 20:01:50.742175 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:50.742365 (+ 190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":473338,"prepare.queue_time_us":2294784,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":143,"raft.queue_time_us":502141,"raft.run_cpu_time_us":626,"raft.run_wall_time_us":39107,"spinlock_wait_cycles":290458368,"thread_start_us":1484854,"threads_started":3,"wal-append.queue_time_us":544647}
W20260504 20:01:50.745461 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.744673 15877 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.750442 15877 raft_consensus.cc:2804] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:01:50.751163 15877 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:50.751821 15877 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:50.753857 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:50.758035 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 43705.2us
I20260504 20:01:50.769414 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.769639 15877 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:50.771080 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.772276 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.773406 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.773710 15877 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:50.667799 15830 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.796079 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.804159 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.808923 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.809818 12743 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:50.810286 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:50.810963 15730 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:50.811620 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.812893 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.811002 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:50.761732 12863 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:50.802179 12822 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:01:50.819196 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.819885 12825 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:50.860260 12820 raft_consensus.cc:1240] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:50.860574 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.860491 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:50.861166 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.861356 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:50.861680 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.862919 12819 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
W20260504 20:01:50.864161 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 44119.6us
W20260504 20:01:50.865167 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 48814.2us
I20260504 20:01:50.870060 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.791465 15864 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:50.876943 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.880440 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.881285 12825 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:50.890831 12816 raft_consensus.cc:2410] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260504 20:01:50.890733 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:50.891013 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 129406us
W20260504 20:01:50.894995 12791 leader_election.cc:336] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.676s (SENT)
I20260504 20:01:50.897423 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.899302 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:50.900305 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.884769 12716 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:50.902009 12863 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:50.902928 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:50.870107 15841 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.001s sys 0.011s
I20260504 20:01:50.905331 12791 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:50.905730 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:50.900693 12714 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:50.906018 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:50.905018 12821 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
W20260504 20:01:50.913985 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:50.891640 15830 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:50.926515 12742 raft_consensus.cc:2410] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260504 20:01:50.922614 12749 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:50.871814 15647 consensus_queue.cc:1048] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.046s
I20260504 20:01:50.925089 12819 raft_consensus.cc:2376] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4. Re-sending same reply.
W20260504 20:01:50.907073 15841 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s user 0.001s sys 0.013s
I20260504 20:01:50.935019 15396 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:50.935597 15396 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:50.936163 15699 consensus_meta.cc:287] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.567s user 0.017s sys 0.000s
I20260504 20:01:50.936787 15534 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:50.937760 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:50.934897 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:50.945436 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 71830.7us
W20260504 20:01:50.946928 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 66596.5us
W20260504 20:01:50.949026 15886 log.cc:927] Time spent T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260504 20:01:50.943818 15864 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:50.953719 15867 log.cc:927] Time spent T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.859s user 0.009s sys 0.004s
I20260504 20:01:50.957614 12863 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:50.966969 12714 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:50.968575 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 299666us
I20260504 20:01:50.969388 12790 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:50.920712 15563 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:50.921893 15755 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:50.935925 15836 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:50.982239 15627 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:50.982443 15836 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:50.980813 13532 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:50.936165 15396 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:50.992827 12864 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:50.993834 12714 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:50.998442 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.998696 15638 raft_consensus.cc:3055] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:50.999071 15638 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:50.999378 12742 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 5: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:50.998904 12863 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:51.000128 15638 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.001981 12863 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:51.003750 15859 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:51.004279 15859 raft_consensus.cc:697] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:51.002790 15755 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:51.005445 15755 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:51.005977 15755 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:51.006129 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:50.977386 15847 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:51.008121 15847 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:51.008563 15847 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:51.005391 15859 consensus_queue.cc:237] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:50.993422 15836 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:50.993156 15800 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:51.013360 15800 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:51.013761 15800 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:51.016062 15755 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:51.017171 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.023052 15755 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:51.023341 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1838) took 1542 ms. Trace:
I20260504 20:01:51.023800 12746 rpcz_store.cc:276] 0504 20:01:49.480634 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:49.482060 (+ 1426us) service_pool.cc:224] Handling call
0504 20:01:49.482516 (+ 456us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:49.483010 (+ 494us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:49.483055 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:49.483729 (+ 674us) log.cc:844] Serialized 26 byte log entry
0504 20:01:50.100419 (+616690us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:01:50.101976 (+ 1557us) log.cc:844] Serialized 14 byte log entry
0504 20:01:50.138563 (+ 36587us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:50.138667 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:50.952929 (+814262us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:51.022788 (+ 69859us) raft_consensus.cc:1685] finished
0504 20:01:51.022925 (+ 137us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:51.023092 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42918,"prepare.queue_time_us":320,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":46,"raft.queue_time_us":150110,"raft.run_cpu_time_us":478,"raft.run_wall_time_us":21480,"spinlock_wait_cycles":23820544,"thread_start_us":137564,"threads_started":1,"wal-append.queue_time_us":415720}
W20260504 20:01:51.031428 15899 log.cc:927] Time spent T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260504 20:01:51.001569 15638 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:50.994984 12716 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:51.041909 15396 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.054034 12892 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
W20260504 20:01:51.056195 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:51.054632 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 64729.5us
I20260504 20:01:51.056190 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 506) took 1087 ms. Trace:
I20260504 20:01:51.057405 12892 rpcz_store.cc:276] 0504 20:01:49.968567 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:51.053466 (+1084899us) service_pool.cc:224] Handling call
0504 20:01:51.053835 (+ 369us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:51.055932 (+ 2097us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:51.056128 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:51.089002 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:51.058835 15870 consensus_meta.cc:287] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.604s user 0.006s sys 0.007s
W20260504 20:01:51.096009 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 101119us
W20260504 20:01:51.096359 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 114747us
I20260504 20:01:51.089459 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:51.101833 15755 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:51.102617 15755 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.103920 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.105176 15730 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:51.105472 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.102901 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.077616 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.075589 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.076356 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:51.058928 15386 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:51.114038 12896 raft_consensus.cc:2393] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:51.054068 15396 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:51.076993 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.113828 12893 raft_consensus.cc:2376] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4. Re-sending same reply.
I20260504 20:01:51.115228 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:51.123692 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.125005 12898 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:01:51.125317 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.132195 15179 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:51.113253 15825 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:51.055971 15563 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.165153 12747 raft_consensus.cc:1275] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260504 20:01:51.118604 15386 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:51.115372 15651 consensus_queue.cc:1059] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260504 20:01:51.126279 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.127595 15823 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:51.129844 15009 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:51.123775 15827 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:51.127466 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.128319 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:51.118669 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.133380 13499 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:51.159883 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.147181 15793 raft_consensus.cc:3055] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:51.148589 12742 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:51.149580 12714 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:51.169772 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.163941 15179 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.172144 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.175767 12892 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:51.105866 15730 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.113917 15870 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.177762 15827 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.185973 15009 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.186494 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.187454 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:51.191756 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.187965 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:51.194603 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:51.196223 15690 raft_consensus.cc:3055] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:51.196784 15690 raft_consensus.cc:740] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:51.190232 15651 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:51.198360 15651 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:51.198796 15651 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:51.198066 15690 consensus_queue.cc:260] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.199592 12898 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:01:51.200008 15690 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:51.201073 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.201843 12899 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:01:51.176119 15793 raft_consensus.cc:740] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:51.205063 15847 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.169258 15386 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:51.204887 15793 consensus_queue.cc:260] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.206223 15793 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:51.210150 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.211191 15563 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:51.211800 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 78527.4us
I20260504 20:01:51.212383 15836 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:51.212648 13499 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:51.191648 15696 consensus_meta.cc:287] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.770s user 0.015s sys 0.000s
I20260504 20:01:51.214572 15836 raft_consensus.cc:697] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:51.224268 12747 raft_consensus.cc:2393] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 2: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:51.228523 15870 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:51.229390 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 106820us
I20260504 20:01:51.229627 12716 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:51.175215 15755 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:51.235580 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.233762 12891 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 8.
I20260504 20:01:51.244362 12714 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:51.245335 12818 raft_consensus.cc:1275] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260504 20:01:51.245688 12818 pending_rounds.cc:85] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 1
I20260504 20:01:51.245959 12818 pending_rounds.cc:107] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260504 20:01:51.246199 12818 raft_consensus.cc:2889] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260504 20:01:51.237190 15698 consensus_meta.cc:287] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.749s user 0.014s sys 0.000s
I20260504 20:01:51.248020 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.249881 15009 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:51.254957 15834 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:51.255400 15179 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:51.255842 15691 raft_consensus.cc:2804] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:51.256290 15691 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:51.256637 15691 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:51.250212 15827 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:51.254611 15730 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:51.258638 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.235443 15847 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:51.263662 12822 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:51.269778 15651 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.271432 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:51.248742 15833 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:51.282176 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.282603 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:51.288575 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.291195 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:51.293463 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:51.293342 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.295400 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:51.301630 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.296401 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:51.306851 13532 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260504 20:01:51.305977 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 336674us
I20260504 20:01:51.295581 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.318652 12899 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:51.328392 12896 raft_consensus.cc:2393] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 2: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:51.316205 12747 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:51.333303 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:51.333648 12897 raft_consensus.cc:1240] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:51.334071 12824 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:51.334751 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.335110 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:51.335963 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:51.336604 12897 raft_consensus.cc:2376] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6. Re-sending same reply.
I20260504 20:01:51.338214 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:51.339095 12819 raft_consensus.cc:2410] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260504 20:01:51.340193 12819 raft_consensus.cc:1240] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:51.341775 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:51.342516 13526 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:01:51.343360 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:51.343984 15887 log.cc:927] Time spent T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.220s user 0.003s sys 0.000s
I20260504 20:01:51.344810 12893 raft_consensus.cc:1240] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:51.346073 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:51.348474 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.349949 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.336226 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:51.351979 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:51.367012 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.377151 12822 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:01:51.382787 12864 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:51.305274 15483 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:01:51.386240 12817 raft_consensus.cc:1275] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260504 20:01:51.390530 12817 pending_rounds.cc:85] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 4
I20260504 20:01:51.390867 12817 pending_rounds.cc:107] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260504 20:01:51.391193 12817 raft_consensus.cc:2889] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:01:51.315549 15888 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:51.394615 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:51.400098 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:51.414636 13499 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:51.418969 12899 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:01:51.299137 15800 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.389880 15827 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:01:51.443964 12819 raft_consensus.cc:1240] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:51.453907 15642 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:01:51.405160 15888 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:51.454579 15642 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:51.454738 15888 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:51.455046 15642 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:51.414062 15841 consensus_queue.cc:1059] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260504 20:01:51.457791 15638 raft_consensus.cc:3055] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:51.405387 15179 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260504 20:01:51.458235 15638 raft_consensus.cc:740] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:51.459393 15638 consensus_queue.cc:260] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.460664 15800 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:51.405319 15836 consensus_queue.cc:237] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.461465 12893 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8. Ops: []
W20260504 20:01:51.454205 15891 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.330s user 0.003s sys 0.000s
I20260504 20:01:51.444654 15827 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:51.465628 15651 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:51.466285 15827 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:51.468936 15638 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:51.471055 15901 log.cc:927] Time spent T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.243s user 0.000s sys 0.002s
I20260504 20:01:51.472388 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:51.483361 13497 leader_election.cc:336] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.677s (SENT)
I20260504 20:01:51.484125 13497 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:51.496908 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:51.497673 12822 raft_consensus.cc:2393] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 8: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:51.510633 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.519502 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:51.521093 12819 raft_consensus.cc:1275] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260504 20:01:51.524644 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.499008 15483 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:51.532904 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:51.533785 12819 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:51.515827 15699 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:51.518393 15841 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:51.513484 15823 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:51.545208 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 65806.9us
W20260504 20:01:51.548911 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.549786 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:51.552872 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:51.556968 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:51.558200 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.561290 12747 raft_consensus.cc:1240] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:51.561836 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:51.562399 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:51.565285 12822 raft_consensus.cc:1275] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260504 20:01:51.572140 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:51.573019 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.578622 12791 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:51.573385 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 90100.2us
I20260504 20:01:51.583362 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.546487 15483 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.584304 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:51.544368 15691 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.571928 15793 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:51.551535 15699 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.593178 12863 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:51.602036 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.603585 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.590024 15870 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:01:51.569077 15823 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.573917 15396 consensus_queue.cc:1048] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260504 20:01:51.606513 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:51.595065 15696 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:51.598090 15793 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:51.589454 15800 consensus_queue.cc:1059] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260504 20:01:51.622464 15696 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:51.625948 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.613883 13497 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:51.613930 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:51.599246 15563 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:51.631700 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.634450 13528 raft_consensus.cc:2368] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:51.637207 15823 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:51.638001 15642 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.641390 15642 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:51.643288 13497 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:51.644225 15914 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:51.645053 15914 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:51.645310 13497 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:51.607813 15179 consensus_queue.cc:1059] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260504 20:01:51.647861 15800 raft_consensus.cc:3055] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:51.664438 15800 raft_consensus.cc:740] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:51.634723 15870 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260504 20:01:51.645429 15914 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:51.665329 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:51.678578 12742 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
W20260504 20:01:51.678820 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 65134.6us
I20260504 20:01:51.682027 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:51.682403 15386 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:51.668372 15711 consensus_queue.cc:1048] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
I20260504 20:01:51.693920 12898 raft_consensus.cc:2368] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5.
I20260504 20:01:51.694339 15699 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:51.663463 15627 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:51.695675 15627 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.685235 12863 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:51.621244 15793 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:01:51.702101 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.698717 15627 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:51.703200 15913 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:51.710937 15696 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:51.711798 15691 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:51.714314 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 112164us
I20260504 20:01:51.715245 12791 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:51.716020 15696 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:51.717852 15696 consensus_queue.cc:1059] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:01:51.719012 15696 raft_consensus.cc:3055] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 20:01:51.719401 15696 raft_consensus.cc:740] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:51.719105 15627 consensus_queue.cc:1048] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:51.720261 15696 consensus_queue.cc:260] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.721518 15696 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 1 FOLLOWER]: Advancing to term 2
I20260504 20:01:51.699342 15914 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:51.725008 15711 consensus_queue.cc:1059] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:01:51.726850 15730 raft_consensus.cc:3055] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:51.727480 15730 raft_consensus.cc:740] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:51.728729 15730 consensus_queue.cc:260] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.734411 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:51.704195 15179 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260504 20:01:51.707787 15907 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260504 20:01:51.771379 15730 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
W20260504 20:01:51.775501 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 130239us
I20260504 20:01:51.943706 12820 raft_consensus.cc:1275] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260504 20:01:51.947320 15914 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:51.952675 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.958062 15889 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:51.958330 15888 consensus_meta.cc:287] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.503s user 0.012s sys 0.011s
I20260504 20:01:51.736342 15800 consensus_queue.cc:260] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:51.961862 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:51.978073 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:51.978626 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:51.979009 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:51.980731 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.987838 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:51.988593 12896 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:51.990314 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.990658 13497 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:51.975782 15827 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.506s user 0.026s sys 0.000s
W20260504 20:01:51.997982 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:51.974606 15870 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:52.012377 13497 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:51.977993 15889 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:51.986320 15696 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:52.019163 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:51.981340 15605 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:51.974772 15009 consensus_queue.cc:1048] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260504 20:01:51.980840 15888 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:52.039952 15888 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:52.041005 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:52.067961 12791 leader_election.cc:336] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.847s (SENT)
I20260504 20:01:52.002050 15827 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:52.006955 15642 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:52.002115 15755 consensus_queue.cc:1048] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
W20260504 20:01:52.117491 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.019693 15889 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:52.139416 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:52.158607 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.170523 12894 raft_consensus.cc:2393] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 9: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:52.171772 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1765) took 1124 ms. Trace:
I20260504 20:01:52.018026 15870 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:52.172317 12746 rpcz_store.cc:276] 0504 20:01:51.047675 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:51.059032 (+ 11357us) service_pool.cc:224] Handling call
0504 20:01:51.080368 (+ 21336us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:51.097071 (+ 16703us) raft_consensus.cc:1534] Early marking committed up to index 4
0504 20:01:51.239415 (+142344us) log.cc:844] Serialized 14 byte log entry
0504 20:01:51.466947 (+227532us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:51.473241 (+ 6294us) log.cc:844] Serialized 26 byte log entry
0504 20:01:51.473425 (+ 184us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:51.473465 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:51.473541 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:52.171461 (+697920us) raft_consensus.cc:1685] finished
0504 20:01:52.171493 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:52.171678 (+ 185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16086,"prepare.queue_time_us":168733,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":25981,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":5311,"thread_start_us":267880,"threads_started":2,"wal-append.queue_time_us":254776}
I20260504 20:01:52.077694 15642 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:52.024638 15696 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:52.292172 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 224244us
I20260504 20:01:52.200706 15642 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:52.287599 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 275039us
I20260504 20:01:52.233999 15605 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:52.334642 15793 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.626s user 0.017s sys 0.001s
I20260504 20:01:52.335259 15793 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:52.336508 12821 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 7. Current term is 8. Ops: []
I20260504 20:01:52.338645 15793 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:52.355556 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.305013 15380 raft_consensus.cc:2749] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:01:52.364336 15483 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:52.385051 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.293867 15914 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:52.389708 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 46434.4us
W20260504 20:01:52.395761 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.401616 12896 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
W20260504 20:01:52.427528 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:52.428545 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.472497 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:52.474269 12817 raft_consensus.cc:3060] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 9
W20260504 20:01:52.499217 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.538609 15699 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:52.538781 15793 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:52.539535 15793 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:52.539434 15699 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:52.542546 12790 leader_election.cc:400] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:52.543052 12790 leader_election.cc:403] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260504 20:01:52.543694 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 131387us
W20260504 20:01:52.551309 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 525851us
W20260504 20:01:52.591718 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.592825 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:52.596506 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:52.601830 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:52.612247 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:52.650676 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 51400.5us
I20260504 20:01:52.627079 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:52.631580 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.639528 12746 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 7. Current term is 8. Ops: []
I20260504 20:01:52.644452 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1776) took 1414 ms. Trace:
I20260504 20:01:52.638542 12747 raft_consensus.cc:1240] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:52.652194 12899 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:52.640950 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:52.637146 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 2100) took 1260 ms. Trace:
I20260504 20:01:52.524603 15827 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:52.652916 12744 rpcz_store.cc:276] 0504 20:01:51.230227 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:51.230426 (+ 199us) service_pool.cc:224] Handling call
0504 20:01:51.240407 (+ 9981us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:51.248603 (+ 8196us) raft_consensus.cc:1534] Early marking committed up to index 3
0504 20:01:51.487375 (+238772us) log.cc:844] Serialized 14 byte log entry
0504 20:01:52.643942 (+1156567us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:52.644009 (+ 67us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:52.644095 (+ 86us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:52.644198 (+ 103us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:52.644359 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1063961,"prepare.queue_time_us":34089,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":272941,"threads_started":2,"wal-append.queue_time_us":206405}
W20260504 20:01:52.640909 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.777967 12892 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
W20260504 20:01:52.759526 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:52.759466 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.795558 12820 rpcz_store.cc:276] 0504 20:01:51.377088 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:51.377302 (+ 214us) service_pool.cc:224] Handling call
0504 20:01:51.414505 (+ 37203us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:51.966925 (+552420us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:52.636858 (+669933us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54000d3c54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:52.637061 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853590784}
I20260504 20:01:52.778915 12898 raft_consensus.cc:1217] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Deduplicated request from leader. Original: 4.3->[6.4-6.4] Dedup: 6.4->[]
I20260504 20:01:52.797273 12898 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260504 20:01:52.780629 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:52.776791 12895 raft_consensus.cc:1240] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260504 20:01:52.778096 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:52.799212 12896 raft_consensus.cc:1240] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 1. Current term is 2. Ops: []
I20260504 20:01:52.799347 12895 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 8. Ops: [6.4-6.4]
I20260504 20:01:52.794099 12891 raft_consensus.cc:1240] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260504 20:01:52.760457 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:52.753420 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:52.722909 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 184685us
I20260504 20:01:52.806766 12816 raft_consensus.cc:1240] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:52.807868 13532 raft_consensus.cc:2393] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 9: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
W20260504 20:01:52.709508 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.779958 12893 raft_consensus.cc:1240] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260504 20:01:52.709728 12750 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:01:52.660343 13499 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:52.550877 15483 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:01:52.830349 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 170109us
I20260504 20:01:52.815979 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:52.832461 13528 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:52.641686 15672 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:52.835638 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:52.650043 15800 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:52.641204 15825 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:52.638470 15684 raft_consensus.cc:3055] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:52.846024 12890 raft_consensus.cc:1240] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260504 20:01:52.846160 12892 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:52.838673 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:52.660240 15834 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:52.706401 15605 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:52.849539 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:52.784375 15699 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:52.791962 15696 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:52.856586 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:52.875177 12894 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 525) took 1205 ms. Trace:
I20260504 20:01:52.861825 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:52.865091 12741 raft_consensus.cc:2393] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 7: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:52.638206 15912 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:52.875798 12894 rpcz_store.cc:276] 0504 20:01:51.661529 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:51.681657 (+ 20128us) service_pool.cc:224] Handling call
0504 20:01:52.865387 (+1183730us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54002cc9d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:52.865596 (+ 209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080762624}
I20260504 20:01:52.876504 12820 raft_consensus.cc:2410] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260504 20:01:52.877206 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:52.880877 12892 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:52.887636 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:52.857579 15684 raft_consensus.cc:740] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:52.855305 12895 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d"
dest_uuid: "732c3560a688494196d22a1a48f89968"
from {username='slave'} at 127.0.0.1:37318
I20260504 20:01:52.874109 15924 raft_consensus.cc:2804] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:01:52.853390 15672 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:52.860760 15825 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:52.986228 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:52.987221 12742 raft_consensus.cc:2393] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 8: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:01:52.987710 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 92111.5us
W20260504 20:01:52.958103 12791 leader_election.cc:336] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.937s (SENT)
W20260504 20:01:52.991978 12817 consensus_meta.cc:287] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.517s user 0.012s sys 0.006s
I20260504 20:01:52.993309 12817 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 9.
I20260504 20:01:52.990645 12791 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:52.964385 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:52.966220 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:52.967171 12897 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:52.967512 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:53.002180 12899 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:52.967881 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.004307 12896 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:52.973220 12895 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:01:52.921236 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:52.909397 15912 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:52.966531 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.008145 12893 raft_consensus.cc:2368] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:53.006887 12890 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:53.008544 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:53.009002 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.006207 12895 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:53.015627 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.016132 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.016952 12747 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:53.019290 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:53.020293 12896 raft_consensus.cc:2368] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:53.015451 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:53.023520 12749 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:53.024442 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.027884 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:52.974050 15932 log.cc:927] Time spent T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.270s user 0.002s sys 0.000s
W20260504 20:01:52.967490 15925 log.cc:927] Time spent T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.262s user 0.002s sys 0.000s
W20260504 20:01:52.968624 15938 log.cc:927] Time spent T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.265s user 0.000s sys 0.001s
W20260504 20:01:52.976275 15939 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.271s user 0.001s sys 0.000s
I20260504 20:01:52.917356 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:53.043459 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:52.975966 15672 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:52.959134 15647 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:53.025321 12750 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:52.931073 15833 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:52.956964 15870 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:52.959159 15698 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:53.032423 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:52.975063 15834 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:52.970932 15386 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:52.978179 15825 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:52.974004 15924 raft_consensus.cc:697] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:53.052333 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.076478 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.039108 15823 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:53.078313 12741 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:01:53.113474 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.084883 15647 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:53.116171 15924 consensus_queue.cc:237] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:53.085817 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.041872 15864 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:53.079902 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:53.087427 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.041000 15543 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:53.081429 15833 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:52.973598 15684 consensus_queue.cc:260] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:53.088405 15642 consensus_meta.cc:287] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.750s user 0.013s sys 0.000s
I20260504 20:01:53.093463 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.088230 15698 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:53.100718 12749 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
W20260504 20:01:53.104269 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 146194us
I20260504 20:01:53.106424 13528 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:01:53.112229 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.113852 12747 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:01:53.086920 15834 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:53.115465 15823 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:53.133692 12893 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:53.118260 15925 log.cc:927] Time spent T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260504 20:01:53.132416 12890 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:53.150624 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.157210 12899 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
W20260504 20:01:53.161208 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.164661 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.165180 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.166674 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2068) took 1169 ms. Trace:
I20260504 20:01:53.171907 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.158428 15684 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:53.167271 12824 rpcz_store.cc:276] 0504 20:01:51.996798 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:51.997113 (+ 315us) service_pool.cc:224] Handling call
0504 20:01:51.997461 (+ 348us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:51.997861 (+ 400us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:51.997894 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:51.998464 (+ 570us) log.cc:844] Serialized 26 byte log entry
0504 20:01:52.342368 (+343904us) raft_consensus.cc:1645] Marking committed up to 1
0504 20:01:52.342411 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:52.342493 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:52.973363 (+630870us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:53.158885 (+185522us) raft_consensus.cc:1685] finished
0504 20:01:53.158912 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:53.166571 (+ 7659us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":926,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":17304,"spinlock_wait_cycles":289193600,"thread_start_us":269928,"threads_started":1,"wal-append.queue_time_us":681662}
W20260504 20:01:53.174090 12791 leader_election.cc:336] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 3.232s (SENT)
I20260504 20:01:53.177680 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:53.155000 15870 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:53.143052 15642 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:53.143370 15864 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:53.192849 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.194027 12899 raft_consensus.cc:2368] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5.
W20260504 20:01:53.196096 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 320186us
I20260504 20:01:53.157375 15543 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:53.212852 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.203408 15940 log.cc:927] Time spent T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260504 20:01:53.189038 12893 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4318d305513d467c8254d7c054ef993e"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:01:53.218124 12893 raft_consensus.cc:606] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:53.218683 12893 raft_consensus.cc:614] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260504 20:01:53.223706 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.224483 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.224965 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.225409 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.225862 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.226294 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.233328 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 199821us
W20260504 20:01:53.235244 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 65305.3us
I20260504 20:01:53.240505 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:53.241607 12824 raft_consensus.cc:2393] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 9: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:53.242615 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:53.243647 12817 raft_consensus.cc:2368] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 7. Current term is 8.
I20260504 20:01:53.191021 15870 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:53.205420 15647 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:53.235548 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.254823 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:53.255671 12824 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 9.
I20260504 20:01:53.252843 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:53.234406 15699 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:53.262184 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.265342 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:53.262851 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 232975us
I20260504 20:01:53.250174 13526 raft_consensus.cc:2393] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 7: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:53.271433 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:53.272399 12825 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:53.273698 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48774 (request call id 338) took 1535 ms. Trace:
I20260504 20:01:53.275934 13497 leader_election.cc:400] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 7. Current term is 8.
I20260504 20:01:53.276422 13497 leader_election.cc:403] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260504 20:01:53.276804 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 244057us
I20260504 20:01:53.278014 12714 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:53.274253 12822 rpcz_store.cc:276] 0504 20:01:51.737880 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:51.738000 (+ 120us) service_pool.cc:224] Handling call
0504 20:01:51.948178 (+210178us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:51.959812 (+ 11634us) raft_consensus.cc:1534] Early marking committed up to index 8
0504 20:01:52.016519 (+ 56707us) log.cc:844] Serialized 14 byte log entry
0504 20:01:52.457484 (+440965us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:52.546969 (+ 89485us) log.cc:844] Serialized 26 byte log entry
0504 20:01:52.547209 (+ 240us) raft_consensus.cc:1645] Marking committed up to 9
0504 20:01:52.547523 (+ 314us) log.cc:844] Serialized 14 byte log entry
0504 20:01:52.571962 (+ 24439us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:52.572107 (+ 145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:52.966786 (+394679us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:01:53.273393 (+306607us) raft_consensus.cc:1685] finished
0504 20:01:53.273431 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:53.273616 (+ 185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51810,"prepare.queue_time_us":503434,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":92,"raft.queue_time_us":54959,"raft.run_cpu_time_us":553,"raft.run_wall_time_us":75905,"spinlock_wait_cycles":233149696,"thread_start_us":482524,"threads_started":2,"wal-append.queue_time_us":674960}
I20260504 20:01:53.287247 12714 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:53.289393 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 280501us
I20260504 20:01:53.298565 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.279897 15933 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 7. Current term is 8.
I20260504 20:01:53.282475 15912 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:53.284364 15690 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:01:53.297446 15698 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:53.342814 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:53.352034 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 110898us
W20260504 20:01:53.363588 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.302824 15651 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260504 20:01:53.376132 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.298863 15823 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:53.377441 12893 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 7: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:53.359251 12890 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:01:53.288789 15847 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
W20260504 20:01:53.388182 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.535303 12895 consensus_meta.cc:287] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.517s user 0.013s sys 0.000s
W20260504 20:01:53.518088 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 153737us
I20260504 20:01:53.532368 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:53.676407 12791 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:53.506161 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 63872.9us
W20260504 20:01:53.611330 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.610579 12750 raft_consensus.cc:1275] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260504 20:01:53.446429 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.583138 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:53.679209 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.632676 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.554203 12892 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:53.653347 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.608217 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:53.591303 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.611474 13528 raft_consensus.cc:1240] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:53.508138 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:53.376168 12863 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:53.683741 12890 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:53.629011 12896 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:01:53.684779 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 308746us
I20260504 20:01:53.510311 12746 raft_consensus.cc:1275] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260504 20:01:53.594617 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 286933us
W20260504 20:01:53.686955 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 372750us
I20260504 20:01:53.510298 12749 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:01:53.682902 12790 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:53.676123 12895 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:53.308022 15833 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:53.671826 15954 log.cc:927] Time spent T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
W20260504 20:01:53.609191 15947 log.cc:927] Time spent T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260504 20:01:53.751005 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:53.745533 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.359150 15690 raft_consensus.cc:697] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:01:53.751789 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 75498.7us
W20260504 20:01:53.843636 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 45011.1us
I20260504 20:01:53.803575 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:53.844249 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:53.751377 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:53.675841 15946 log.cc:927] Time spent T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.090s user 0.002s sys 0.000s
W20260504 20:01:53.531338 15937 log.cc:927] Time spent T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
I20260504 20:01:53.756318 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:53.750814 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:53.781447 12895 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:01:53.662839 15483 consensus_meta.cc:287] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.825s user 0.019s sys 0.000s
I20260504 20:01:53.751067 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.380827 15698 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:53.445385 15638 consensus_queue.cc:1059] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.391s
I20260504 20:01:53.806149 15827 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:53.643215 15870 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:53.586110 15951 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:53.535578 15943 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:53.836441 12817 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:53.543493 15647 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:53.914206 15647 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:53.914374 15943 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:53.916730 15647 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:53.689587 15386 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:53.918200 15386 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:53.918462 15951 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:53.920786 15386 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:53.921526 15870 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:53.692927 15711 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.144s
W20260504 20:01:53.446045 15936 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.284s user 0.001s sys 0.001s
I20260504 20:01:53.380589 15945 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:53.522897 15543 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:53.693564 15859 raft_consensus.cc:2804] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:01:53.837380 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2079) took 1504 ms. Trace:
I20260504 20:01:53.469046 15864 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:53.547462 15944 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:53.790140 15834 consensus_meta.cc:287] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.635s user 0.015s sys 0.000s
W20260504 20:01:53.779861 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 52440.9us
I20260504 20:01:53.866437 15959 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:01:53.805881 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.917610 15943 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:53.813917 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.802958 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.789970 15962 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:53.554842 15934 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:53.839108 12865 connection.cc:254] RPC call timeout handler was delayed by -1.4753868699999657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15600000000000003s for a response
I20260504 20:01:53.845216 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:53.791636 12746 raft_consensus.cc:1275] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:53.445331 15642 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:53.846995 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:53.853410 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:53.856536 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.974237 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.974691 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.975021 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.975387 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:53.975725 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:53.830802 15825 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:53.865578 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:37318 (request call id 1774) took 1072 ms. Trace:
I20260504 20:01:53.743382 15957 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:53.850576 15833 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:53.894240 15690 consensus_queue.cc:237] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:53.861783 15483 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:01:53.865293 15698 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:53.903304 12823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2073) took 1589 ms. Trace:
W20260504 20:01:53.908416 15684 consensus_meta.cc:287] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.727s user 0.011s sys 0.002s
I20260504 20:01:53.924039 15963 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:54.003288 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 159125us
I20260504 20:01:53.921435 15956 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:53.441139 15933 consensus_queue.cc:1059] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.396s
I20260504 20:01:53.948285 12894 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:01:54.004185 12824 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:54.014863 12894 raft_consensus.cc:606] T 43dc05259beb4569a29aeb5327c9b3bc P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:53.970690 15859 raft_consensus.cc:697] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [term 9 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:01:53.956045 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 273196us
I20260504 20:01:53.964076 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:53.963893 12816 rpcz_store.cc:276] 0504 20:01:52.333005 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:52.829317 (+496312us) service_pool.cc:224] Handling call
0504 20:01:52.832620 (+ 3303us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:52.851368 (+ 18748us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:52.851442 (+ 74us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:52.864503 (+ 13061us) log.cc:844] Serialized 26 byte log entry
0504 20:01:53.227305 (+362802us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:53.227389 (+ 84us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:53.227496 (+ 107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:53.837131 (+609635us) raft_consensus.cc:1685] finished
0504 20:01:53.837153 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:53.837310 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9209,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":7425,"thread_start_us":278606,"threads_started":1,"wal-append.queue_time_us":464166}
I20260504 20:01:54.028244 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:53.980175 12820 raft_consensus.cc:2393] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:53.970523 15945 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260504 20:01:53.953198 15870 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.029726 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:53.977310 12895 rpcz_store.cc:276] 0504 20:01:52.834150 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:52.846637 (+ 12487us) service_pool.cc:224] Handling call
0504 20:01:53.850155 (+1003518us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:53.986212 12865 connection.cc:254] RPC call timeout handler was delayed by -1.4222868699999935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
W20260504 20:01:54.038200 12865 connection.cc:254] RPC call timeout handler was delayed by -1.4060868699999673s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260504 20:01:54.038776 12865 connection.cc:254] RPC call timeout handler was delayed by -1.3925868699999455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20260504 20:01:54.039227 12865 connection.cc:254] RPC call timeout handler was delayed by -1.3880868699999382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
W20260504 20:01:54.039623 12865 connection.cc:254] RPC call timeout handler was delayed by -1.2422868699999299s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20260504 20:01:54.039708 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:54.003671 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2070) took 2003 ms. Trace:
I20260504 20:01:54.010511 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:54.040840 13532 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:01:54.010412 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:54.003950 12823 rpcz_store.cc:276] 0504 20:01:52.313925 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:52.331984 (+ 18059us) service_pool.cc:224] Handling call
0504 20:01:52.359941 (+ 27957us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:52.360383 (+ 442us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:52.360420 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:52.361086 (+ 666us) log.cc:844] Serialized 26 byte log entry
0504 20:01:52.793061 (+431975us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:52.793103 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:52.793188 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:53.902922 (+1109734us) raft_consensus.cc:1685] finished
0504 20:01:53.902969 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:53.903184 (+ 215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":525,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":128,"thread_start_us":166555,"threads_started":1,"wal-append.queue_time_us":670680}
I20260504 20:01:53.971067 15959 raft_consensus.cc:697] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:53.948606 12817 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:53.922758 15827 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.013605 13499 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:54.040938 12818 rpcz_store.cc:276] 0504 20:01:52.000115 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:52.000241 (+ 126us) service_pool.cc:224] Handling call
0504 20:01:52.003343 (+ 3102us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:52.005358 (+ 2015us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:52.005399 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:52.005774 (+ 375us) log.cc:844] Serialized 26 byte log entry
0504 20:01:52.406638 (+400864us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:52.406738 (+ 100us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:52.406948 (+ 210us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:54.003394 (+1596446us) raft_consensus.cc:1685] finished
0504 20:01:54.003423 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:54.003601 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":12677,"spinlock_wait_cycles":27904,"thread_start_us":228682,"threads_started":1,"wal-append.queue_time_us":1001041}
W20260504 20:01:54.040089 12865 connection.cc:254] RPC call timeout handler was delayed by -1.136986869999987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
I20260504 20:01:54.041797 12791 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:53.851163 15961 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:53.967375 15834 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.053926 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:54.030755 12820 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
W20260504 20:01:54.051301 13497 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.003s sys 0.011s
W20260504 20:01:54.058241 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 44743.1us
W20260504 20:01:54.063988 13497 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.003s sys 0.011s
I20260504 20:01:54.065661 12790 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:54.045490 15483 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:54.003325 15833 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:54.073516 12716 leader_election.cc:336] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.862s (SENT)
I20260504 20:01:54.076800 12716 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:54.029475 15945 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:54.003487 15825 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.080042 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:54.052774 15961 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:54.091372 15924 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.019s sys 0.020s
I20260504 20:01:54.125049 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:54.128268 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 54805.9us
W20260504 20:01:54.128327 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 63440.8us
I20260504 20:01:54.108121 15933 raft_consensus.cc:3055] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Stepping down as leader of term 2
W20260504 20:01:54.120242 15924 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s user 0.020s sys 0.020s
W20260504 20:01:54.135093 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 69536.4us
W20260504 20:01:54.135855 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:54.113232 15961 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:54.114054 15698 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:54.145057 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 89158.4us
I20260504 20:01:54.078166 15483 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:54.150540 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 70581.1us
W20260504 20:01:54.153131 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:54.153719 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:54.154479 12895 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:01:54.155158 13497 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:54.148700 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:54.159426 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 101809us
I20260504 20:01:54.152874 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:54.154505 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:54.163148 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:01:54.160467 12863 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:54.165284 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:54.142964 15968 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:01:54.169202 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:54.174214 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:54.174912 12747 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:01:54.138991 15933 raft_consensus.cc:740] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:54.176003 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:54.175820 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:54.177722 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:54.178635 12742 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:54.175112 12750 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:54.147760 15951 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:54.190493 12716 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:01:54.196103 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:54.207358 12865 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:54.183594 15543 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:01:54.209748 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:54.174422 15968 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:54.180562 15973 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:54.217723 12715 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:54.223172 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 62786.1us
I20260504 20:01:54.224167 12863 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:01:54.224852 12865 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:54.209695 15951 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:54.127914 15859 consensus_queue.cc:237] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.192320 15870 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.192462 15698 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:01:54.210078 15543 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:54.234980 15698 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:54.235805 15971 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260504 20:01:54.238880 15698 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:54.239446 15483 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:54.239705 15969 log.cc:927] Time spent T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260504 20:01:54.231531 15924 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:01:54.218667 15827 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:54.236426 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 51878.5us
I20260504 20:01:54.211321 15967 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:54.221582 15968 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:54.250515 12716 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:54.254611 12716 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:54.242604 15711 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:54.257247 15910 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:01:54.257359 15793 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.110869 15945 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.258188 15910 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:54.041756 15956 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.014179 15963 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.262473 15833 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.145900 15959 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.322310 15831 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:54.325268 12823 raft_consensus.cc:1275] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:01:54.327466 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:54.263092 15823 consensus_queue.cc:1059] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:01:54.232925 15642 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:53.978492 15934 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.237717 15651 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:54.263938 15827 raft_consensus.cc:2804] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:54.250689 15967 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:54.007925 15957 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.342623 15823 raft_consensus.cc:3055] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:54.343276 15823 raft_consensus.cc:740] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:54.343842 15957 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:53.969043 15944 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.344316 15823 consensus_queue.cc:260] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.346057 15944 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:54.346167 15823 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:54.347422 15944 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:54.347992 15944 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.275494 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:54.276685 15825 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.243609 15933 consensus_queue.cc:260] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.277485 15847 consensus_queue.cc:1059] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:01:53.979784 15962 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.237790 15951 raft_consensus.cc:697] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:54.278178 15672 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:54.232613 15949 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:54.243098 15834 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.278990 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 2107) took 1468 ms. Trace:
W20260504 20:01:54.290004 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 124843us
W20260504 20:01:54.300631 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:54.242416 15924 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:54.313782 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 96181.5us
W20260504 20:01:54.311565 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 87481.6us
I20260504 20:01:54.236727 15543 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:54.318146 15974 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:01:54.318768 15975 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:54.380918 15975 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:54.381524 15975 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:54.319641 15956 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.319820 15945 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.321343 15963 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.328094 12823 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:54.388882 12816 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:54.336040 12821 raft_consensus.cc:2468] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:01:54.336582 12898 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:54.350154 12825 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:54.354184 12791 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:54.362884 13528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 675) took 1078 ms. Trace:
I20260504 20:01:54.338155 15642 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:54.370846 12819 rpcz_store.cc:276] 0504 20:01:52.810773 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:52.958417 (+147644us) service_pool.cc:224] Handling call
0504 20:01:52.961054 (+ 2637us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:53.156158 (+195104us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:53.156209 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:53.156270 (+ 61us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:53.156329 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:54.278722 (+1122393us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54000e47d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:54.278771 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:54.278917 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":448035712}
I20260504 20:01:54.361363 15944 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.414738 15944 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:54.415983 15933 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:54.395124 15642 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:54.418150 15827 raft_consensus.cc:697] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:54.415660 12825 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:54.419341 15827 consensus_queue.cc:237] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.390349 13528 rpcz_store.cc:276] 0504 20:01:53.271756 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:53.271911 (+ 155us) service_pool.cc:224] Handling call
0504 20:01:53.611260 (+339349us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:53.681739 (+ 70479us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:54.349992 (+668253us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54004902d0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:54.350151 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779705600}
I20260504 20:01:54.360728 15967 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:54.424438 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 199649us
I20260504 20:01:54.426322 12865 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:54.398483 12714 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:54.405416 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:54.406522 15951 consensus_queue.cc:237] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.407265 15986 consensus_queue.cc:1048] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260504 20:01:54.408519 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 54440.1us
I20260504 20:01:54.411191 15962 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.413679 12822 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 7. Current term is 8. Ops: []
I20260504 20:01:54.385124 15924 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Advancing to term 10
W20260504 20:01:54.448830 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 47825.3us
I20260504 20:01:54.442227 12791 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:54.453720 13499 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:54.460525 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 62137.4us
I20260504 20:01:54.418051 15543 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.489413 12714 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:54.494459 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 52334.3us
I20260504 20:01:54.497494 15690 consensus_queue.cc:1059] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260504 20:01:54.387600 15672 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.500236 15934 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:54.503983 13499 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:54.473093 15647 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:54.510843 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 119557us
I20260504 20:01:54.512876 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:54.494257 15962 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:54.524366 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:54.519281 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:54.519716 13499 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:54.486277 15958 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:54.477478 15991 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:54.540859 15991 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:54.521536 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1787) took 1494 ms. Trace:
I20260504 20:01:54.523638 12894 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:01:54.542654 12791 leader_election.cc:400] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5.
I20260504 20:01:54.542994 12791 leader_election.cc:403] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:01:54.502223 15672 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:54.525895 13498 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:54.512271 15864 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:01:54.533484 15543 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.538354 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:54.541706 12896 rpcz_store.cc:276] 0504 20:01:53.027353 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:53.027570 (+ 217us) service_pool.cc:224] Handling call
0504 20:01:54.521398 (+1493828us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b540013b0d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:54.521480 (+ 82us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":596577664}
I20260504 20:01:54.535431 15698 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:54.549921 15985 log.cc:927] Time spent T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20260504 20:01:54.549525 15642 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.556476 15825 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:54.557097 15825 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.557698 13528 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:54.558494 15642 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.558619 15647 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:01:54.559335 15825 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:54.536278 15962 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:54.561316 15962 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:54.561755 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:54.550383 15996 raft_consensus.cc:3055] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:54.574019 15996 raft_consensus.cc:740] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:54.575114 15996 consensus_queue.cc:260] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.510527 15981 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:54.560725 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:54.566921 12816 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:54.571715 15961 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.576074 15864 raft_consensus.cc:697] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:54.552364 15966 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:54.589318 15698 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:54.554127 15945 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5.
I20260504 20:01:54.546967 15991 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:01:54.591346 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 48840.8us
I20260504 20:01:54.582180 15981 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:54.647452 15981 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:54.594821 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:57644 (request call id 54) took 1808 ms. Trace:
I20260504 20:01:54.595150 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:01:54.596400 15994 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260504 20:01:54.650653 13530 rpcz_store.cc:276] 0504 20:01:52.786349 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:52.805522 (+ 19173us) service_pool.cc:224] Handling call
0504 20:01:54.594103 (+1788581us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54006de554. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c918470 00007f015c9b731a 00007f016198611e 00007f015af02437 00007f015af023c2 00007f015af022f1 00007f015af0226c 00007f015af01522 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:54.594734 (+ 631us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3027553536}
I20260504 20:01:54.599215 15996 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:54.641865 12791 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:54.659504 12898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1785) took 1693 ms (client timeout 2066 ms). Trace:
W20260504 20:01:54.659989 12898 rpcz_store.cc:269] 0504 20:01:52.983297 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:53.008185 (+ 24888us) service_pool.cc:224] Handling call
0504 20:01:54.659289 (+1651104us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400207bd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:54.659424 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2323559296}
W20260504 20:01:54.644255 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 139187us
I20260504 20:01:54.589304 15924 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:54.645485 15864 consensus_queue.cc:237] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.651700 12896 raft_consensus.cc:2393] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:01:54.589641 12821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 340) took 1351 ms (client timeout 1581 ms). Trace:
I20260504 20:01:54.601089 15973 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:01:54.669523 12821 rpcz_store.cc:269] 0504 20:01:53.238744 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:53.239118 (+ 374us) service_pool.cc:224] Handling call
0504 20:01:54.589412 (+1350294us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b54000e3dd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:54.589561 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1709505280}
I20260504 20:01:54.628464 15966 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:54.674014 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:54.688472 12865 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:54.693203 13526 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 9. Current term is 10. Ops: []
W20260504 20:01:54.704151 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 196595us
I20260504 20:01:54.705013 12714 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:54.711673 12821 raft_consensus.cc:1240] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:54.697388 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:54.714730 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:54.714807 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:54.715734 12819 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:54.672492 15945 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:54.677906 15966 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:54.711282 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:54.743839 12895 raft_consensus.cc:1240] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:54.746985 15924 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:54.753445 15960 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:54.754066 15960 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:54.754429 15960 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:54.690809 15696 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260504 20:01:54.761083 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:54.768209 16007 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
W20260504 20:01:54.773766 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 58397.1us
I20260504 20:01:54.804162 12823 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:01:54.680374 15968 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.751091 15973 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260504 20:01:54.826254 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 135908us
I20260504 20:01:54.854760 12790 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:54.875835 15698 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:54.876367 15698 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:54.877115 15698 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:54.886585 15961 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:54.893388 15992 log.cc:927] Time spent T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260504 20:01:54.801981 15605 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:54.900462 12750 raft_consensus.cc:1240] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 2. Current term is 3. Ops: []
I20260504 20:01:54.803325 15945 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:55.007414 15933 consensus_meta.cc:287] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.591s user 0.012s sys 0.000s
I20260504 20:01:54.965451 15958 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:54.940593 15967 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:54.968494 15945 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.169011 12817 raft_consensus.cc:1240] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:55.228557 15973 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:55.229241 15973 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:55.230494 16012 log.cc:927] Time spent T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20260504 20:01:55.238993 15973 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:55.264669 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:55.280977 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 592614us
W20260504 20:01:55.282400 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 585093us
W20260504 20:01:55.294661 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 467141us
I20260504 20:01:55.304945 12791 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:55.306629 15968 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:55.268086 15945 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:55.309753 13497 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:55.313380 12894 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.021s sys 0.035s
I20260504 20:01:54.898631 15698 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:55.326059 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 44034.4us
I20260504 20:01:55.327126 12865 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:55.319258 12894 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s user 0.022s sys 0.035s
I20260504 20:01:55.319159 12750 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 7. Current term is 8. Ops: []
I20260504 20:01:55.321133 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:01:55.322381 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:55.296634 15975 consensus_meta.cc:287] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.914s user 0.008s sys 0.007s
W20260504 20:01:55.332439 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 48966.7us
I20260504 20:01:55.335105 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:55.335402 12824 raft_consensus.cc:1240] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:55.339655 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:55.340611 15543 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:55.347164 15698 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:55.348520 12865 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:55.350198 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:55.351337 12865 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:55.352859 15605 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:55.355247 15987 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 20:01:55.356050 15987 raft_consensus.cc:2764] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260504 20:01:55.348680 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:55.363230 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:55.364069 15975 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:55.366426 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 70669.2us
I20260504 20:01:55.353590 15605 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:01:55.365172 15949 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.026s sys 0.052s
I20260504 20:01:55.325778 15968 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:55.378612 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:55.381678 12818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.050s sys 0.080s
I20260504 20:01:55.383882 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:55.391084 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:55.391424 12818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.051s sys 0.080s
I20260504 20:01:55.391825 12742 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:55.391873 12896 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:01:55.392419 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.391108 12823 raft_consensus.cc:2368] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 5. Current term is 6.
I20260504 20:01:55.385782 12825 raft_consensus.cc:1275] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260504 20:01:55.366001 15962 consensus_meta.cc:287] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.797s user 0.012s sys 0.002s
I20260504 20:01:55.390313 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:55.392584 12896 raft_consensus.cc:606] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:01:55.386689 12824 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 8. Current term is 9. Ops: []
W20260504 20:01:55.390419 15949 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s user 0.027s sys 0.053s
I20260504 20:01:55.361626 16007 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260504 20:01:55.409653 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.409873 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.410599 12742 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:55.413452 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.413933 12821 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260504 20:01:55.414366 12742 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:01:55.416201 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.416970 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.372191 16003 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260504 20:01:55.421487 12747 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:55.424793 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2093) took 1396 ms. Trace:
I20260504 20:01:55.425312 12820 rpcz_store.cc:276] 0504 20:01:54.028133 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:54.111696 (+ 83563us) service_pool.cc:224] Handling call
0504 20:01:54.127876 (+ 16180us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:54.334085 (+206209us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:54.334126 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:54.334179 (+ 53us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:54.334224 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:55.424511 (+1090287us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54000a4bd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:55.424572 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:55.424720 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473599744}
I20260504 20:01:55.426376 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:55.402861 15864 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.022s sys 0.047s
I20260504 20:01:55.421834 12741 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 2.
W20260504 20:01:55.410882 15981 consensus_meta.cc:287] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.763s user 0.014s sys 0.000s
W20260504 20:01:55.447963 12791 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 2.066s (SENT) [suppressed 49 similar messages]
I20260504 20:01:55.470417 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1787) took 1790 ms. Trace:
I20260504 20:01:55.427104 12746 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:55.480789 12743 rpcz_store.cc:276] 0504 20:01:53.637359 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:53.637578 (+ 219us) service_pool.cc:224] Handling call
0504 20:01:55.423964 (+1786386us) spinlock_profiling.cc:243] Waited 698 ms on lock 0x7b5400012054. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:55.427137 (+ 3173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1606219008}
I20260504 20:01:55.483784 15975 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:55.455210 15864 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s user 0.023s sys 0.047s
I20260504 20:01:55.399400 15962 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:55.500059 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:55.499255 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:55.506525 12743 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 9: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:55.476596 15967 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:55.527989 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.529323 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.530849 12747 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:55.480832 15981 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.534253 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:55.535003 12746 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:55.538108 12744 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260504 20:01:55.497886 15543 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:55.548359 12822 raft_consensus.cc:1240] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:55.501564 15912 consensus_queue.cc:1059] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260504 20:01:55.560149 15543 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.605733 12741 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 8: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:01:55.606817 15386 raft_consensus.cc:3055] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:01:55.607311 15386 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:55.605999 15965 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:01:55.604442 16008 log.cc:927] Time spent T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.177s user 0.001s sys 0.001s
I20260504 20:01:55.600481 15981 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:55.619930 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 133353us
W20260504 20:01:55.608445 16013 log.cc:927] Time spent T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
W20260504 20:01:55.620815 12716 leader_election.cc:336] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.503s (SENT)
W20260504 20:01:55.621230 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 173322us
W20260504 20:01:55.625592 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 126445us
I20260504 20:01:55.624512 12716 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:55.634105 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:55.639783 12791 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:55.641094 12791 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:55.642697 15912 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:55.644692 12823 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 7. Current term is 8. Ops: []
W20260504 20:01:55.642762 12716 leader_election.cc:336] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.552s (SENT)
I20260504 20:01:55.645642 12716 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:55.648247 15943 raft_consensus.cc:3055] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:01:55.648747 15943 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:55.649994 15698 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:55.649685 15943 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.650851 15543 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:55.651234 15943 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:55.651608 15651 raft_consensus.cc:3055] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:01:55.652041 15651 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:55.653106 12791 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:55.652889 15651 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:55.612049 15965 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.655802 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2102) took 1030 ms. Trace:
I20260504 20:01:55.663060 12791 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:55.664916 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:55.668524 12791 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:55.667627 12818 rpcz_store.cc:276] 0504 20:01:54.625105 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:54.625325 (+ 220us) service_pool.cc:224] Handling call
0504 20:01:54.625790 (+ 465us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:54.626405 (+ 615us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:54.626447 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:54.666101 (+ 39654us) log.cc:844] Serialized 26 byte log entry
0504 20:01:55.392094 (+725993us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:55.392139 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:55.392206 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:01:55.655544 (+263338us) raft_consensus.cc:1685] finished
0504 20:01:55.655569 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:55.655727 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":119,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":17451,"thread_start_us":668405,"threads_started":1,"wal-append.queue_time_us":734710}
W20260504 20:01:55.669335 12716 leader_election.cc:336] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.566s (SENT)
I20260504 20:01:55.669835 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:55.671480 12716 leader_election.cc:336] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.580s (SENT)
I20260504 20:01:55.672326 12716 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:55.678483 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 325940us
I20260504 20:01:55.670183 12791 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:55.678076 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 342058us
W20260504 20:01:55.681114 12716 leader_election.cc:336] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.628s (SENT)
I20260504 20:01:55.681823 12716 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:55.717212 12865 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:55.718391 12716 leader_election.cc:336] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.638s (SENT)
I20260504 20:01:55.719043 12716 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:55.724072 12865 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:55.729760 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:55.730474 15962 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:55.730738 15651 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:55.731292 15962 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:01:55.731731 15962 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:55.732123 15962 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:01:55.655725 15698 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:55.733103 15698 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:55.733502 15698 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:55.736137 15975 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:01:55.736593 15996 consensus_meta.cc:287] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.068s user 0.008s sys 0.005s
I20260504 20:01:55.737107 15975 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:55.664343 15834 consensus_queue.cc:1059] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260504 20:01:55.728049 15958 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:55.739080 15834 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:55.608321 15386 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:55.740207 15958 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:55.740837 15949 raft_consensus.cc:3055] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:01:55.716145 15543 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:55.744174 15958 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:55.747316 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:55.679813 15534 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260504 20:01:55.750005 16020 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260504 20:01:55.715865 15945 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:55.766206 15945 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:55.715487 15959 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:01:55.767793 15945 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:55.768031 15959 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:55.770054 15949 raft_consensus.cc:740] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:55.737712 15996 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:55.723306 15912 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:01:55.666869 15967 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:55.765213 15934 consensus_queue.cc:1048] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.002s
I20260504 20:01:55.792140 13528 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 9. Current term is 10. Ops: []
I20260504 20:01:55.794916 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:55.795733 13526 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
W20260504 20:01:55.797243 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 50284.2us
I20260504 20:01:55.797385 12715 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:55.764263 15973 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:55.798453 12865 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:55.798887 15973 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.799767 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:55.800361 15827 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:55.800707 16006 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260504 20:01:55.801235 15973 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:55.799304 15982 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:01:55.803705 12865 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:55.769296 15959 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:55.775489 15983 consensus_queue.cc:1048] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
W20260504 20:01:55.823129 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 203563us
W20260504 20:01:55.814368 15966 consensus_meta.cc:287] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.067s user 0.008s sys 0.009s
I20260504 20:01:55.830965 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:55.832072 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:55.802003 15967 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:01:55.832348 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:55.834240 12816 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 2092) took 1806 ms (client timeout 1655 ms). Trace:
W20260504 20:01:55.838969 12816 rpcz_store.cc:269] 0504 20:01:54.027848 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:54.028005 (+ 157us) service_pool.cc:224] Handling call
0504 20:01:55.834034 (+1806029us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54003acfd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:55.834161 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":827835648}
I20260504 20:01:55.840682 12865 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:01:55.810992 15981 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:55.844673 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:55.845733 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 209621us
I20260504 20:01:55.820549 15912 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:01:55.850458 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.851883 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:55.863420 13528 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:01:55.854395 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:55.864645 13530 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:55.857179 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:55.856693 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 186576us
I20260504 20:01:55.871124 12791 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:55.871568 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 153234us
I20260504 20:01:55.872380 12791 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:55.872884 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.838837 15982 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:55.877753 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.880285 12892 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:55.843035 15958 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:55.844887 15967 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:55.877470 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.883031 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.847839 15975 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:55.824172 15996 consensus_queue.cc:1059] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260504 20:01:55.884254 13497 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:01:55.895694 12716 leader_election.cc:336] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.814s (SENT)
I20260504 20:01:55.890323 12816 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:55.898159 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:55.898885 12741 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:55.848093 15711 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:01:55.904374 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:55.900843 12716 leader_election.cc:336] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.826s (SENT)
I20260504 20:01:55.905367 12716 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:55.862659 15912 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:55.773396 15949 consensus_queue.cc:260] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:55.872212 15943 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:55.902614 12746 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 9.
I20260504 20:01:55.842238 16007 consensus_queue.cc:1048] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
I20260504 20:01:55.906533 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.910503 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:55.914752 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.918030 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 540) took 1402 ms. Trace:
I20260504 20:01:55.842629 15966 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.920316 12891 raft_consensus.cc:1275] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260504 20:01:55.923930 12716 leader_election.cc:336] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.845s (SENT)
I20260504 20:01:55.930141 12716 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:55.923506 12895 rpcz_store.cc:276] 0504 20:01:54.505767 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:54.505951 (+ 184us) service_pool.cc:224] Handling call
0504 20:01:54.506282 (+ 331us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:54.751572 (+245290us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:55.907835 (+1156263us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54001b9c54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:55.908098 (+ 263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":545540480}
I20260504 20:01:55.864261 15981 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.938191 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.942483 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:01:55.943403 12746 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:01:55.946360 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:55.922339 15960 consensus_meta.cc:287] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.104s user 0.019s sys 0.001s
I20260504 20:01:55.947116 12893 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:01:55.915997 15912 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:55.950719 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.952450 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.882344 15605 consensus_queue.cc:1059] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260504 20:01:55.907465 16029 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:01:55.960893 16029 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:55.961432 16029 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:55.961550 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.948624 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.964118 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.964946 13532 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:01:55.965206 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:55.963001 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:55.966945 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.968145 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.968779 12895 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:55.969625 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.970283 12895 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:55.974519 15847 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:55.975914 15847 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:55.976428 15847 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:55.976750 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 92593.1us
I20260504 20:01:55.978227 12790 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:55.979324 15605 raft_consensus.cc:3055] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Stepping down as leader of term 8
I20260504 20:01:55.979828 15605 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:55.925391 16024 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:55.981935 12750 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 8. Current term is 9. Ops: []
I20260504 20:01:55.992715 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:55.993149 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.995846 12895 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 5.
I20260504 20:01:55.997721 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:56.000685 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:56.001310 12741 raft_consensus.cc:2468] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:01:56.001716 12750 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:55.949385 15973 raft_consensus.cc:2764] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260504 20:01:56.010659 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:55.886598 15982 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:55.992512 15605 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:56.006904 16027 log.cc:927] Time spent T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260504 20:01:56.007113 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.031111 12899 raft_consensus.cc:2393] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:55.917472 16006 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260504 20:01:56.014478 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:56.032300 12895 raft_consensus.cc:2368] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 2. Current term is 3.
W20260504 20:01:56.008559 15991 consensus_meta.cc:287] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.342s user 0.019s sys 0.000s
W20260504 20:01:56.011318 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 103246us
I20260504 20:01:56.044124 12749 raft_consensus.cc:1240] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260504 20:01:56.044507 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:56.045447 13528 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:55.949662 16018 consensus_queue.cc:1059] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260504 20:01:56.052476 12897 raft_consensus.cc:2393] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:56.052701 12715 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:56.021653 12746 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:55.929101 15943 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:55.954340 16030 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:55.994515 15949 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:56.018059 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 102313us
I20260504 20:01:56.068543 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.068768 12791 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:56.063504 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.008229 15847 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:55.966977 15960 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.028831 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:01:56.074816 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.078292 12895 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:56.079576 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.079979 12791 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:56.086177 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 162300us
I20260504 20:01:56.047605 15965 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:56.087062 12716 leader_election.cc:336] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.848s (SENT)
I20260504 20:01:56.087647 12716 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:56.086203 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.070940 15834 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:56.070259 16007 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:56.097357 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:56.091893 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:56.090278 12716 leader_election.cc:336] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Timed out: RequestConsensusVote RPC to 127.11.68.3:39151 timed out after 1.937s (SENT)
I20260504 20:01:56.098613 12716 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:01:56.098407 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.101083 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:56.101756 12816 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:56.003633 16024 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.098357 12744 raft_consensus.cc:2468] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:56.097095 15834 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:56.128913 16041 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:56.107919 16007 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.130014 16041 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:56.130504 16041 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:56.132180 16007 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:56.133517 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.134990 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.112393 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:56.135689 12895 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:01:56.111722 12716 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:56.103374 12890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 534) took 2374 ms. Trace:
I20260504 20:01:56.109537 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.138732 15834 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:56.137179 12890 rpcz_store.cc:276] 0504 20:01:53.729194 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:53.729340 (+ 146us) service_pool.cc:224] Handling call
0504 20:01:53.790837 (+ 61497us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:54.367142 (+576305us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:54.367176 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:54.367232 (+ 56us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:01:54.367273 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:56.103143 (+1735870us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54002cc4d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:56.103198 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:56.103305 (+ 107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1324746496}
I20260504 20:01:56.144053 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.136184 12824 raft_consensus.cc:2376] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3. Re-sending same reply.
I20260504 20:01:56.083163 15605 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:56.051563 15991 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.087857 16037 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:01:56.165309 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:56.149102 13526 raft_consensus.cc:2393] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 8: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:01:56.085597 16036 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:01:56.170881 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:56.166177 12890 raft_consensus.cc:2376] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7. Re-sending same reply.
I20260504 20:01:56.169936 16037 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:56.136772 15698 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.159509 16034 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260504 20:01:56.169356 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 89475.2us
I20260504 20:01:56.180941 12893 raft_consensus.cc:2368] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
W20260504 20:01:56.235270 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 148669us
W20260504 20:01:56.200156 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 283023us
I20260504 20:01:56.182615 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.250885 12791 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:56.261233 13497 leader_election.cc:336] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.581s (SENT)
I20260504 20:01:56.249624 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:01:56.167780 15996 raft_consensus.cc:2749] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:01:56.298481 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.314815 12890 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:01:56.316143 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.317724 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:56.319228 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:56.216290 15960 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:56.307807 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 318361us
W20260504 20:01:56.310264 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 60823.2us
I20260504 20:01:56.320962 12890 raft_consensus.cc:3060] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:56.304440 13530 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
W20260504 20:01:56.242326 16026 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20260504 20:01:56.326532 12790 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:01:56.237476 15651 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:56.328678 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.329664 12895 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:01:56.200299 15966 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:56.248745 15534 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:01:56.220907 15982 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:56.288252 16034 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:56.293062 15981 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:56.235098 16037 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:01:56.291847 15968 consensus_meta.cc:287] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.898s user 0.018s sys 0.000s
I20260504 20:01:56.334712 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.338194 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:56.363389 12895 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:01:56.347116 15996 consensus_queue.cc:1059] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:56.336030 12865 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:56.178198 15912 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:56.383785 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:56.383836 15912 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:56.386530 16024 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:56.388790 16045 log.cc:927] Time spent T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260504 20:01:56.389153 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:01:56.419777 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 83964.9us
I20260504 20:01:56.369138 16041 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.370591 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.372012 15996 raft_consensus.cc:3055] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:56.372455 15963 consensus_queue.cc:1059] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260504 20:01:56.377930 13530 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 9. Current term is 10. Ops: []
I20260504 20:01:56.378624 16049 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:01:56.380860 15968 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:01:56.346542 16034 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:56.320173 16043 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.074s
W20260504 20:01:56.402467 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:01:56.453405 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:56.340114 16042 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:01:56.339699 15943 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:56.465765 12890 raft_consensus.cc:2468] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 9.
I20260504 20:01:56.483135 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.484228 12890 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 8: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:01:56.476164 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 55363.6us
I20260504 20:01:56.485574 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:01:56.485060 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 192692us
W20260504 20:01:56.510298 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:56.531340 12819 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
W20260504 20:01:56.556005 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 206020us
I20260504 20:01:56.341922 15651 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.461697 13526 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:01:56.464181 15996 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:56.467568 15831 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:01:56.381625 16029 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.474788 16048 consensus_queue.cc:1059] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.235s
I20260504 20:01:56.596580 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:56.679762 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 353293us
I20260504 20:01:56.817741 12821 raft_consensus.cc:2393] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:56.593551 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:01:56.637198 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 48885.8us
I20260504 20:01:56.671806 12895 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:56.591951 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:01:56.585664 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 100214us
I20260504 20:01:56.478307 15963 consensus_queue.cc:1243] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer 732c3560a688494196d22a1a48f89968 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260504 20:01:56.597009 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 485338us
I20260504 20:01:56.835873 12716 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:56.616218 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 365413us
I20260504 20:01:56.582795 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.819700 13499 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
W20260504 20:01:56.701179 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 101765us
I20260504 20:01:56.823280 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:01:56.838313 12893 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:01:56.838763 12816 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:01:56.851696 12791 connection.cc:254] RPC call timeout handler was delayed by -1.2728912430000037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
I20260504 20:01:56.818833 12824 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:01:56.862455 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.887360 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:01:56.684607 15605 consensus_meta.cc:287] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.524s user 0.008s sys 0.003s
I20260504 20:01:56.561825 16057 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:01:56.815012 16056 log.cc:927] Time spent T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
W20260504 20:01:56.880532 15962 consensus_meta.cc:287] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.134s user 0.021s sys 0.040s
W20260504 20:01:56.726868 16053 log.cc:927] Time spent T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
I20260504 20:01:56.815676 16043 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:56.889266 12750 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:01:56.896260 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:01:56.898365 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.900769 12864 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:01:56.953012 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 52348.2us
I20260504 20:01:56.903061 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:01:56.706663 16045 log.cc:927] Time spent T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260504 20:01:56.697113 16058 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:01:56.589066 15968 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:56.821022 15831 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:56.918610 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:56.917758 12816 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:01:56.706231 15991 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:56.923391 12899 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 9.
I20260504 20:01:56.890483 12895 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:01:57.010987 12865 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:01:56.904907 15982 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:01:56.925552 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 105963us
I20260504 20:01:56.926416 16064 raft_consensus.cc:2804] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:01:56.935009 15996 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:56.936293 16041 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:56.937729 16049 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:56.558432 16039 consensus_queue.cc:1059] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.375s
I20260504 20:01:56.938968 15386 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:01:56.981961 16042 raft_consensus.cc:697] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:56.881119 15698 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:56.924642 16057 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.006899 15831 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:01:57.017119 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:57.039384 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:57.054636 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:57.096772 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1807) took 1044 ms. Trace:
I20260504 20:01:57.097270 12895 rpcz_store.cc:276] 0504 20:01:56.052770 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.010993 (+958223us) service_pool.cc:224] Handling call
0504 20:01:57.096678 (+ 85685us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:01:57.056748 12741 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.015s sys 0.034s
W20260504 20:01:57.098942 12741 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s user 0.015s sys 0.034s
I20260504 20:01:56.979751 15605 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260504 20:01:57.061208 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 320971us
I20260504 20:01:56.974061 12747 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 7. Current term is 8. Ops: []
W20260504 20:01:57.068930 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 209555us
W20260504 20:01:57.072220 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 133647us
W20260504 20:01:57.100152 12791 connection.cc:254] RPC call timeout handler was delayed by -1.1144912429999749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1744s for a response
I20260504 20:01:57.073863 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:57.102859 12816 raft_consensus.cc:2393] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:01:57.075125 12825 raft_consensus.cc:1240] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:57.104085 12890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1810) took 1050 ms. Trace:
I20260504 20:01:56.953042 15962 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.104491 12890 rpcz_store.cc:276] 0504 20:01:56.053606 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.994239 (+940633us) service_pool.cc:224] Handling call
0504 20:01:57.104018 (+109779us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:57.009555 16046 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:57.076304 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:01:57.123921 12824 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:01:57.104508 12825 raft_consensus.cc:1240] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:57.040535 15968 raft_consensus.cc:697] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:01:57.119692 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:57.063684 16041 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:57.122864 13530 raft_consensus.cc:1275] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260504 20:01:57.103652 15945 consensus_meta.cc:287] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.319s user 0.014s sys 0.001s
I20260504 20:01:57.082351 16064 raft_consensus.cc:697] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
W20260504 20:01:57.136336 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 112395us
I20260504 20:01:56.984867 16043 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:01:57.143669 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:57.161160 12822 raft_consensus.cc:1240] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:01:57.146756 12825 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 7. Current term is 8. Ops: []
W20260504 20:01:57.153884 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 143016us
I20260504 20:01:57.109359 15996 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:01:57.119426 15698 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:57.159034 12824 raft_consensus.cc:1240] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
I20260504 20:01:57.157253 12895 raft_consensus.cc:2376] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6. Re-sending same reply.
I20260504 20:01:57.169523 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1811) took 1115 ms. Trace:
I20260504 20:01:57.060595 15982 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.173606 16058 raft_consensus.cc:697] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:57.177055 16059 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:57.177428 16058 consensus_queue.cc:237] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.177690 16059 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:01:57.182305 16054 log.cc:927] Time spent T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260504 20:01:57.082417 16049 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.187618 13532 raft_consensus.cc:3055] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:57.188150 13532 raft_consensus.cc:740] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:01:57.190682 15859 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:01:57.191282 15859 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.150616 15651 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:57.195413 15834 consensus_meta.cc:287] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.056s user 0.010s sys 0.003s
I20260504 20:01:57.204857 16057 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:01:57.206549 16072 log.cc:927] Time spent T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260504 20:01:57.207340 15962 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.221069 16029 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.222208 15968 consensus_queue.cc:237] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.201416 16043 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.122236 15647 consensus_queue.cc:1048] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260504 20:01:57.161550 16080 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:57.212380 13532 consensus_queue.cc:260] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.234082 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:57.315135 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2088) took 3288 ms. Trace:
I20260504 20:01:57.172971 12895 rpcz_store.cc:276] 0504 20:01:56.062958 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.110809 (+1047851us) service_pool.cc:224] Handling call
0504 20:01:57.169371 (+ 58562us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:57.154903 15945 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.156718 16041 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.315745 12817 rpcz_store.cc:276] 0504 20:01:54.026743 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:54.054941 (+ 28198us) service_pool.cc:224] Handling call
0504 20:01:54.070664 (+ 15723us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:55.169675 (+1099011us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:57.314890 (+2145215us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003002d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:57.315067 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2525202688}
W20260504 20:01:57.322139 12791 connection.cc:254] RPC call timeout handler was delayed by -1.06589124300001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
I20260504 20:01:57.170684 16006 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:01:57.324257 16069 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260504 20:01:57.324555 16075 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:01:57.185732 16049 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:57.347802 16075 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.348502 16049 consensus_queue.cc:1059] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:01:57.141705 16046 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.351090 16075 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:57.317349 16003 raft_consensus.cc:3055] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Stepping down as leader of term 9
I20260504 20:01:57.351907 16046 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:57.352157 16003 raft_consensus.cc:740] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:01:57.353525 15859 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:57.353756 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:57.328264 15982 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:57.329630 13525 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260504 20:01:57.330192 16064 consensus_queue.cc:237] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.334616 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:01:57.336220 15959 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.506s user 0.012s sys 0.004s
I20260504 20:01:57.360680 13524 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:01:57.362072 15959 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.353780 15945 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.364996 15959 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.365518 16041 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:57.367441 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:58902 (request call id 10) took 1984 ms. Trace:
I20260504 20:01:57.367357 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:57.368005 12896 rpcz_store.cc:276] 0504 20:01:55.376432 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.377254 (+ 822us) service_pool.cc:224] Handling call
0504 20:01:57.359947 (+1982693us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b5400290ca0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c92b900 00007f015c9a8a32 00007f0161984cca 00007f015aefd934 00007f015aefd8c2 00007f015aefd7f1 00007f015aefd76c 00007f015aefca22 00007f015ace8bcc 00007f015ace8006
0504 20:01:57.360251 (+ 304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2693376}
I20260504 20:01:57.369238 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:57.369401 12864 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:01:57.369727 13526 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.675s user 0.007s sys 0.003s
I20260504 20:01:57.370281 13526 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
W20260504 20:01:57.338863 16034 consensus_meta.cc:287] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.781s user 0.011s sys 0.000s
I20260504 20:01:57.354693 15988 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:01:57.371381 16034 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.371490 15988 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.317265 16039 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:01:57.342701 12816 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:01:57.373553 16040 consensus_queue.cc:1059] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260504 20:01:57.344035 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:57.374104 16039 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.375092 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1818) took 1309 ms. Trace:
I20260504 20:01:57.375584 16034 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.318163 16080 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.375509 12896 rpcz_store.cc:276] 0504 20:01:56.065795 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.368934 (+1303139us) service_pool.cc:224] Handling call
0504 20:01:57.375019 (+ 6085us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:57.377537 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:57.378386 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2065) took 1437 ms. Trace:
I20260504 20:01:57.378762 12896 rpcz_store.cc:276] 0504 20:01:55.940517 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.377191 (+1436674us) service_pool.cc:224] Handling call
0504 20:01:57.378325 (+ 1134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:57.346614 12791 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:01:57.355481 12897 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2031) took 1505 ms (client timeout 1685 ms). Trace:
W20260504 20:01:57.380139 12897 rpcz_store.cc:269] 0504 20:01:55.850045 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.850175 (+ 130us) service_pool.cc:224] Handling call
0504 20:01:57.355279 (+1505104us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b540013b0d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:01:57.355411 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":424992000}
W20260504 20:01:57.333647 16037 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.948s user 0.006s sys 0.003s
I20260504 20:01:57.357513 12825 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260504 20:01:57.343262 15967 consensus_meta.cc:287] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.434s user 0.015s sys 0.000s
I20260504 20:01:57.175940 15698 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.273957 16042 consensus_queue.cc:237] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.374527 12895 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:01:57.235582 15949 consensus_meta.cc:287] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.161s user 0.010s sys 0.004s
I20260504 20:01:57.323416 16059 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:57.375763 15831 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.344946 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:57.217729 15834 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:01:57.273101 15958 consensus_queue.cc:1059] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
W20260504 20:01:57.511734 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:01:57.514647 15989 raft_consensus.cc:3055] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:01:57.515300 15989 raft_consensus.cc:740] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:01:57.515897 15831 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:57.516309 12894 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 2024) took 1672 ms. Trace:
I20260504 20:01:57.516602 15989 consensus_queue.cc:260] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.516817 12894 rpcz_store.cc:276] 0504 20:01:55.843714 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.843915 (+ 201us) service_pool.cc:224] Handling call
0504 20:01:55.844334 (+ 419us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:55.919819 (+ 75485us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:55.919867 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:01:55.919928 (+ 61us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:55.919987 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:01:57.515980 (+1595993us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b54000f4e54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:01:57.516043 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:01:57.516194 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172907264}
I20260504 20:01:57.519160 13528 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:01:57.468849 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:57.384938 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:57.408608 12892 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:01:57.384790 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:01:57.523152 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2053) took 1608 ms. Trace:
W20260504 20:01:57.509146 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 147565us
I20260504 20:01:57.523581 12897 rpcz_store.cc:276] 0504 20:01:55.914754 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.384628 (+1469874us) service_pool.cc:224] Handling call
0504 20:01:57.523086 (+138458us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:01:57.343636 16006 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:01:57.525859 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:57.526775 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:01:57.376371 16083 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:57.535637 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:01:57.350370 16049 raft_consensus.cc:3055] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:01:57.376483 16039 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:01:58.685707 12679 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0f20 after lost signal to thread 11536
W20260504 20:01:59.759761 12895 consensus_meta.cc:287] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 2.241s user 0.018s sys 0.010s
W20260504 20:02:00.690327 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800328540 after lost signal to thread 11539
I20260504 20:02:00.690510 12895 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
W20260504 20:02:00.691717 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1831) took 4617 ms (client timeout 2227 ms). Trace:
W20260504 20:02:00.692247 12895 rpcz_store.cc:269] 0504 20:01:56.074017 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.323312 (+1249295us) service_pool.cc:224] Handling call
0504 20:02:00.691493 (+3368181us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:02:00.694206 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1826) took 4621 ms (client timeout 2747 ms). Trace:
I20260504 20:01:57.532310 15967 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:01:57.449255 16034 raft_consensus.cc:2764] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260504 20:01:57.386893 16037 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:00.698252 12821 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:00.698756 15967 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.469069 16041 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 2.639s
I20260504 20:01:59.829177 16060 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:01:57.390657 16080 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.562829 16083 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:00.703387 16060 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:00.706622 16083 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:01:57.740243 16086 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 20:02:00.708056 16086 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:00.708221 16060 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:00.708901 16086 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 2 FOLLOWER]: Advancing to term 3
I20260504 20:02:00.710915 16037 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:01:57.754114 16091 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:01:58.305671 16095 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20260504 20:01:57.539700 16049 raft_consensus.cc:740] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:00.713125 16091 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:00.679028 15989 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:01:57.523078 15605 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:00.715044 15605 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:00.694612 12895 rpcz_store.cc:269] 0504 20:01:56.072727 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.693875 (+4621148us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.694114 (+ 239us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 20:02:00.717101 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 554) took 3817 ms (client timeout 2057 ms). Trace:
W20260504 20:02:00.719537 12895 rpcz_store.cc:269] 0504 20:01:56.899968 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.716811 (+3816843us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.716973 (+ 162us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 20:02:00.722730 16091 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:00.723063 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 555) took 3822 ms (client timeout 2065 ms). Trace:
W20260504 20:02:00.723693 12895 rpcz_store.cc:269] 0504 20:01:56.900259 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.722823 (+3822564us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.722970 (+ 147us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 20:02:00.724278 12821 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
I20260504 20:02:00.721444 16049 consensus_queue.cc:260] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:00.726193 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1836) took 3794 ms (client timeout 2066 ms). Trace:
W20260504 20:02:00.731232 12895 rpcz_store.cc:269] 0504 20:01:56.932093 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.724608 (+3792515us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.726076 (+ 1468us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 20:02:00.727903 16049 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:00.730260 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:00.734051 12820 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:02:00.735400 12820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 2156) took 4141 ms (client timeout 2316 ms). Trace:
W20260504 20:02:00.732535 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1837) took 3800 ms (client timeout 2124 ms). Trace:
W20260504 20:02:00.738211 12895 rpcz_store.cc:269] 0504 20:01:56.932327 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.732287 (+3799960us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.732462 (+ 175us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 20:02:00.740006 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1838) took 3807 ms (client timeout 2145 ms). Trace:
W20260504 20:02:00.740494 12895 rpcz_store.cc:269] 0504 20:01:56.932544 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.739803 (+3807259us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.739941 (+ 138us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 20:02:00.738751 12820 rpcz_store.cc:269] 0504 20:01:56.593773 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.597064 (+ 3291us) service_pool.cc:224] Handling call
0504 20:02:00.729643 (+4132579us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:02:00.735230 (+ 5587us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25804672}
I20260504 20:02:00.753966 16086 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:00.757350 16086 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:00.759279 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1839) took 3826 ms (client timeout 2217 ms). Trace:
W20260504 20:02:00.769346 12895 rpcz_store.cc:269] 0504 20:01:56.932768 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.758937 (+3826169us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.759180 (+ 243us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 20:02:00.771045 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38968 (request call id 561) took 3535 ms (client timeout 1950 ms). Trace:
W20260504 20:02:00.771606 12895 rpcz_store.cc:269] 0504 20:01:57.235260 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.770704 (+3535444us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.770941 (+ 237us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 20:02:00.772758 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1832) took 4698 ms (client timeout 3188 ms). Trace:
W20260504 20:02:00.773191 12895 rpcz_store.cc:269] 0504 20:01:56.074265 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.772487 (+4698222us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.772660 (+ 173us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 20:02:00.774283 12895 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2050) took 4918 ms (client timeout 4048 ms). Trace:
W20260504 20:02:00.774677 12895 rpcz_store.cc:269] 0504 20:01:55.856194 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.774049 (+4917855us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:00.774207 (+ 158us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 20:02:00.777309 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 2058) took 4838 ms. Trace:
I20260504 20:02:00.777837 12895 rpcz_store.cc:276] 0504 20:01:55.938443 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.775406 (+4836963us) service_pool.cc:224] Handling call
0504 20:02:00.776198 (+ 792us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.776755 (+ 557us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:00.776800 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:00.776876 (+ 76us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:02:00.776958 (+ 82us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.777024 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:00.777179 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.780273 12895 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8. Ops: []
I20260504 20:02:00.781013 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 2059) took 4842 ms. Trace:
I20260504 20:02:00.786021 12895 rpcz_store.cc:276] 0504 20:01:55.938740 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.779247 (+4840507us) service_pool.cc:224] Handling call
0504 20:02:00.779873 (+ 626us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.780708 (+ 835us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.780940 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.788457 12895 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:02:00.790634 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 2062) took 4851 ms. Trace:
I20260504 20:02:00.791081 12895 rpcz_store.cc:276] 0504 20:01:55.939639 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.787151 (+4847512us) service_pool.cc:224] Handling call
0504 20:02:00.787765 (+ 614us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.788966 (+ 1201us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.790516 (+ 1550us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.792974 12895 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:00.793782 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 2063) took 4853 ms. Trace:
I20260504 20:02:00.794179 12895 rpcz_store.cc:276] 0504 20:01:55.939915 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.792013 (+4852098us) service_pool.cc:224] Handling call
0504 20:02:00.792535 (+ 522us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.793450 (+ 915us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.793704 (+ 254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.796543 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 556) took 3896 ms. Trace:
I20260504 20:02:00.784368 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 2060) took 4845 ms. Trace:
W20260504 20:02:00.797480 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 3.27317e+06us
I20260504 20:02:00.797834 12898 rpcz_store.cc:276] 0504 20:01:55.939022 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.782646 (+4843624us) service_pool.cc:224] Handling call
0504 20:02:00.783264 (+ 618us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.783715 (+ 451us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:00.783746 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:00.783811 (+ 65us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:02:00.783923 (+ 112us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.784007 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:00.784145 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.797006 12895 rpcz_store.cc:276] 0504 20:01:56.900519 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.795119 (+3894600us) service_pool.cc:224] Handling call
0504 20:02:00.795674 (+ 555us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.796008 (+ 334us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:00.796045 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:00.796129 (+ 84us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:02:00.796224 (+ 95us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.796295 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:00.796471 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:02:00.800352 12744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1795) took 4845 ms (client timeout 1744 ms). Trace:
W20260504 20:02:00.801191 12744 rpcz_store.cc:269] 0504 20:01:55.954893 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.955555 (+ 662us) service_pool.cc:224] Handling call
0504 20:02:00.799718 (+4844163us) spinlock_profiling.cc:243] Waited 91.4 ms on lock 0x7b5400012554. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:00.799954 (+ 236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":210245504}
I20260504 20:02:00.800566 12898 raft_consensus.cc:1275] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260504 20:02:00.801044 12895 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:02:00.803788 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 2066) took 3870 ms. Trace:
I20260504 20:02:00.804306 12898 rpcz_store.cc:276] 0504 20:01:56.933022 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.799426 (+3866404us) service_pool.cc:224] Handling call
0504 20:02:00.800032 (+ 606us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.803284 (+ 3252us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.803685 (+ 401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22912}
I20260504 20:01:57.521914 12790 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:00.809588 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 558) took 3691 ms. Trace:
I20260504 20:02:00.810102 12898 rpcz_store.cc:276] 0504 20:01:57.117869 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.806792 (+3688923us) service_pool.cc:224] Handling call
0504 20:02:00.807357 (+ 565us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.809160 (+ 1803us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:00.809190 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:00.809259 (+ 69us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:02:00.809294 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.809361 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:00.809515 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.810408 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:02:00.812127 12896 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2068) took 3878 ms (client timeout 1698 ms). Trace:
W20260504 20:02:00.812548 12896 rpcz_store.cc:269] 0504 20:01:56.933542 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.536717 (+603175us) service_pool.cc:224] Handling call
0504 20:02:00.812029 (+3275312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.814967 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 559) took 3691 ms. Trace:
I20260504 20:02:00.815521 12896 rpcz_store.cc:276] 0504 20:01:57.123523 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.813276 (+3689753us) service_pool.cc:224] Handling call
0504 20:02:00.814158 (+ 882us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.814531 (+ 373us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:00.814557 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:00.814619 (+ 62us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:02:00.814694 (+ 75us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.814758 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:00.814882 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:02:00.817080 15996 consensus_meta.cc:287] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 3.649s user 0.021s sys 0.015s
I20260504 20:02:00.818405 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 560) took 3641 ms. Trace:
W20260504 20:02:00.818580 13497 connection.cc:254] RPC call timeout handler was delayed by -3.2648727470000267s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
I20260504 20:02:00.818792 12896 rpcz_store.cc:276] 0504 20:01:57.176907 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.817078 (+3640171us) service_pool.cc:224] Handling call
0504 20:02:00.817602 (+ 524us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.818090 (+ 488us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:00.818125 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:00.818188 (+ 63us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:02:00.818213 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.818258 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:00.818357 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.820716 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 562) took 3510 ms. Trace:
W20260504 20:02:00.820551 13497 leader_election.cc:336] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.711s (SENT)
I20260504 20:02:00.821295 13497 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:00.822530 15933 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260504 20:02:00.824684 13497 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.718s (SENT) [suppressed 13 similar messages]
I20260504 20:02:00.827783 12898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 563) took 3361 ms. Trace:
I20260504 20:02:00.828881 12896 rpcz_store.cc:276] 0504 20:01:57.314659 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.819744 (+3505085us) service_pool.cc:224] Handling call
0504 20:02:00.820174 (+ 430us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.820465 (+ 291us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:00.820485 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:00.820526 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:02:00.820546 (+ 20us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.820580 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:00.820657 (+ 77us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.830433 12898 rpcz_store.cc:276] 0504 20:01:57.466982 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.826607 (+3359625us) service_pool.cc:224] Handling call
0504 20:02:00.827114 (+ 507us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.827373 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:00.827406 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:00.827488 (+ 82us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:02:00.827524 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.827582 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:00.827719 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:00.833714 15933 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:00.834794 15933 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:00.836680 16060 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:00.837693 16060 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:00.842644 13497 connection.cc:254] RPC call timeout handler was delayed by -2.86214733099996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20260504 20:02:00.843240 13497 connection.cc:254] RPC call timeout handler was delayed by -2.8338093180000214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
I20260504 20:02:00.851948 15933 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:00.852365 16060 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:01:57.427096 15988 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:00.857853 13497 connection.cc:254] RPC call timeout handler was delayed by -1.8365847979999899s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.195s for a response
W20260504 20:02:00.858433 13497 connection.cc:254] RPC call timeout handler was delayed by -1.5121687829999928s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.213s for a response
W20260504 20:02:00.858764 13497 connection.cc:254] RPC call timeout handler was delayed by -1.472320320000108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20260504 20:02:00.859071 13497 connection.cc:254] RPC call timeout handler was delayed by -1.4707687830000395s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
I20260504 20:02:00.859926 15698 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:00.871354 15834 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:00.909992 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2131) took 3834 ms. Trace:
I20260504 20:02:00.911098 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2122) took 4810 ms. Trace:
I20260504 20:02:00.911602 12822 rpcz_store.cc:276] 0504 20:01:56.100979 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.101104 (+ 125us) service_pool.cc:224] Handling call
0504 20:01:56.101395 (+ 291us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:57.164440 (+1063045us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.910510 (+3746070us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540015dbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:00.910846 (+ 336us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2425621760}
I20260504 20:02:00.910660 12816 rpcz_store.cc:276] 0504 20:01:57.074886 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.141167 (+ 66281us) service_pool.cc:224] Handling call
0504 20:01:57.143372 (+ 2205us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:57.374129 (+230757us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.909190 (+3535061us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54003ae154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:00.909636 (+ 446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":457914624}
I20260504 20:01:57.537315 16006 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:00.931193 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 412) took 3909 ms. Trace:
I20260504 20:02:00.931728 13530 rpcz_store.cc:276] 0504 20:01:57.021501 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.021691 (+ 190us) service_pool.cc:224] Handling call
0504 20:01:57.082102 (+ 60411us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:57.142543 (+ 60441us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:00.930515 (+3787972us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x7b5400440a54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:00.930903 (+ 388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92877184}
I20260504 20:02:00.932299 16006 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:00.936802 12892 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2052) took 5054 ms (client timeout 1900 ms). Trace:
W20260504 20:01:57.426489 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 79943us
I20260504 20:02:00.940897 16091 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:00.941913 12791 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:00.941833 16091 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:00.942396 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 3.42024e+06us
I20260504 20:02:00.944326 12791 leader_election.cc:400] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 3 election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 3. Current term is 4.
I20260504 20:02:00.944527 16091 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:00.945451 16091 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:00.946204 15973 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:00.947280 16091 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:00.947729 16091 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:02:00.937301 12892 rpcz_store.cc:269] 0504 20:01:55.882581 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.962711 (+ 80130us) service_pool.cc:224] Handling call
0504 20:02:00.936360 (+4973649us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400002b54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:00.936580 (+ 220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3288605440}
I20260504 20:02:00.950845 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:00.951006 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:00.951390 16006 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:00.951741 13527 raft_consensus.cc:2368] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9.
I20260504 20:02:00.951824 13528 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:02:00.941954 16037 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:00.955757 16037 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:00.956223 12897 raft_consensus.cc:2368] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
W20260504 20:02:00.958307 12897 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2067) took 4025 ms (client timeout 1685 ms). Trace:
W20260504 20:02:00.958822 12897 rpcz_store.cc:269] 0504 20:01:56.933300 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.525547 (+592247us) service_pool.cc:224] Handling call
0504 20:02:00.957983 (+3432436us) spinlock_profiling.cc:243] Waited 3.43 s on lock 0x7b540013b0d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:00.958189 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7878006144}
I20260504 20:02:00.961001 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:00.961205 13526 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:02:00.965245 13532 raft_consensus.cc:3060] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:00.966387 16080 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:00.966802 12790 connection.cc:254] RPC call timeout handler was delayed by -1.1319394900000361s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2699s for a response
I20260504 20:02:00.967129 16080 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:00.967923 12790 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:00.968981 12790 leader_election.cc:400] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9.
I20260504 20:02:00.969254 12790 leader_election.cc:403] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260504 20:02:00.970155 16041 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:02:00.970252 16037 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:00.970656 16041 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:00.970979 16037 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) with higher term. Message: Invalid argument: T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9.
I20260504 20:02:00.971154 16041 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:00.978294 12791 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:00.982038 13497 leader_election.cc:336] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.629s (SENT)
I20260504 20:02:00.983551 15967 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:02:00.985020 12899 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1828) took 4911 ms (client timeout 1636 ms). Trace:
W20260504 20:02:00.985549 12899 rpcz_store.cc:269] 0504 20:01:56.073285 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.083242 (+ 9957us) service_pool.cc:224] Handling call
0504 20:02:00.984577 (+4901335us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b5400208ad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:00.984778 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1894029056}
W20260504 20:02:00.990756 13499 connection.cc:254] RPC call timeout handler was delayed by -2.8582679060000373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
W20260504 20:02:00.991340 13499 connection.cc:254] RPC call timeout handler was delayed by -2.7088679060000231s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
W20260504 20:02:00.991652 13499 connection.cc:254] RPC call timeout handler was delayed by -2.5531679059999988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20260504 20:02:00.991961 13499 connection.cc:254] RPC call timeout handler was delayed by -2.5459679059999871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
W20260504 20:02:00.992287 13499 connection.cc:254] RPC call timeout handler was delayed by -2.5459679059999871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
W20260504 20:02:00.992601 13499 connection.cc:254] RPC call timeout handler was delayed by -2.0288938700000472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.195s for a response
W20260504 20:02:00.992939 13499 connection.cc:254] RPC call timeout handler was delayed by -1.6661234310000737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
I20260504 20:02:00.995075 12791 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:00.995867 15981 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:00.996151 16034 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:00.996713 16091 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:00.997977 12791 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:00.996793 15981 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.003286 15967 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:01.003530 15981 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:01.003931 12819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 346) took 5669 ms (client timeout 1629 ms). Trace:
W20260504 20:02:01.004365 12819 rpcz_store.cc:269] 0504 20:01:55.334115 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.334772 (+ 657us) service_pool.cc:224] Handling call
0504 20:02:01.003440 (+5668668us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540039a154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:01.003657 (+ 217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2749260672}
I20260504 20:02:01.009900 16080 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:01.011327 16091 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:01.014263 16034 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:02:01.014304 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:02:01.009655 12821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 2160) took 4414 ms (client timeout 2464 ms). Trace:
I20260504 20:02:01.016134 13527 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:01.013532 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2138) took 3857 ms. Trace:
W20260504 20:02:01.016202 12821 rpcz_store.cc:269] 0504 20:01:56.595225 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.818812 (+223587us) service_pool.cc:224] Handling call
0504 20:01:56.902903 (+ 84091us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:02:01.009387 (+4106484us) spinlock_profiling.cc:243] Waited 3.74 s on lock 0x7b54000e4cd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:01.009563 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8615835392}
I20260504 20:02:01.015753 16034 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:01.018867 16034 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:01.017653 12825 rpcz_store.cc:276] 0504 20:01:57.155612 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:57.164856 (+ 9244us) service_pool.cc:224] Handling call
0504 20:01:57.165128 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:57.381480 (+216352us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:01.012959 (+3631479us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540015e0d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:01.013326 (+ 367us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":441666304}
I20260504 20:02:01.020995 12791 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:02:01.022284 16058 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 3.473s user 0.020s sys 0.047s
I20260504 20:02:01.024570 12791 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:01.024605 16058 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.485s user 0.020s sys 0.048s
I20260504 20:02:01.025686 16080 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:01.032501 15967 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:02:01.039163 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:01.039163 16080 raft_consensus.cc:2804] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:02:01.040835 16080 raft_consensus.cc:697] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:01.040946 13532 raft_consensus.cc:1275] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260504 20:02:01.041043 16091 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:01.041772 13532 pending_rounds.cc:85] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Aborting all ops after (but not including) 6
I20260504 20:02:01.043527 13532 pending_rounds.cc:107] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260504 20:02:01.043834 13532 raft_consensus.cc:2889] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:02:01.044659 16035 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.045342 16035 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:01.046623 16032 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260504 20:02:01.042408 16080 consensus_queue.cc:237] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.050542 12790 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:01.057109 16091 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:02:01.057462 15967 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:01.058014 15967 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:01.058527 15967 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:01.058616 13532 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 410) took 5092 ms. Trace:
I20260504 20:02:01.066329 13532 rpcz_store.cc:276] 0504 20:01:55.966335 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.967729 (+ 1394us) service_pool.cc:224] Handling call
0504 20:01:56.005118 (+ 37389us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:57.187463 (+1182345us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b540043eca0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9261a0 00007f015c9b05a0 00007f015c9b1611 00007f015c9ae16f 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c
0504 20:02:00.964564 (+3777101us) spinlock_profiling.cc:243] Waited 3.65 s on lock 0x7b5800822bd0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c90a48a 00007f015c98c827 00007f015c9a53c7 00007f015c9a6bbe 00007f015c9b0f29 00007f015c9b17f6 00007f015c9ae16f 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34
0504 20:02:01.044368 (+ 79804us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:01.058527 (+ 14159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8826107008}
I20260504 20:02:01.077518 12745 raft_consensus.cc:2410] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260504 20:02:01.073941 13497 leader_election.cc:336] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.130s (SENT)
I20260504 20:02:01.091529 13497 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:01.083781 12894 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2057) took 5145 ms (client timeout 2779 ms). Trace:
W20260504 20:02:01.092569 12894 rpcz_store.cc:269] 0504 20:01:55.938078 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:01.083307 (+5145229us) service_pool.cc:211] Skipping call since client already timed out
0504 20:02:01.083521 (+ 214us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 20:02:01.087088 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37334 (request call id 2030) took 5237 ms. Trace:
I20260504 20:02:01.081372 12791 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:01.094158 12891 rpcz_store.cc:276] 0504 20:01:55.849636 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.849840 (+ 204us) service_pool.cc:224] Handling call
0504 20:01:55.870592 (+ 20752us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:55.929881 (+ 59289us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:01.086361 (+5156480us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b54002085d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:01.086821 (+ 460us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92241280}
I20260504 20:02:01.095556 12791 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:01.096161 16017 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260504 20:02:01.096462 15698 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:01.097180 15698 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:01.097525 15698 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:01.097913 15698 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:01.072409 16041 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.101379 16041 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:01.103242 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:01.105211 12791 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:01.108870 15943 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:02:01.110450 15973 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:02:01.077710 15967 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.111334 15973 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:01.111835 15973 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:01.112251 15973 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:01.113591 15967 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:01.123157 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:01.123560 12791 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:01.093570 13497 leader_election.cc:336] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.176s (SENT)
I20260504 20:02:01.124832 13497 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:01.131078 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.131502 12821 raft_consensus.cc:1240] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:01.131887 12825 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:02:01.133213 16091 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:01.128944 15698 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.049788 16034 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:01.256072 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 132598us
I20260504 20:02:01.273245 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:02:01.370966 13498 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:01.279224 12893 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1825) took 5211 ms (client timeout 2146 ms). Trace:
I20260504 20:02:01.380486 12825 raft_consensus.cc:2393] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:02:01.378427 12679 debug-util.cc:398] Leaking SignalData structure 0x7b08002f3e40 after lost signal to thread 12682
I20260504 20:02:01.222539 16017 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:02:01.262344 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:01.263876 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:02:01.380748 12893 rpcz_store.cc:269] 0504 20:01:56.067760 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.883738 (+815978us) service_pool.cc:224] Handling call
0504 20:02:01.278739 (+4395001us) spinlock_profiling.cc:243] Waited 4.08 s on lock 0x7b540013bad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:01.278972 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9379159296}
I20260504 20:02:01.270078 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1810) took 4395 ms. Trace:
I20260504 20:02:01.385960 12745 rpcz_store.cc:276] 0504 20:01:56.874321 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.874953 (+ 632us) service_pool.cc:224] Handling call
0504 20:02:01.269671 (+4394718us) spinlock_profiling.cc:243] Waited 4.12 s on lock 0x7b54003d9d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:01.269984 (+ 313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9486735104}
W20260504 20:02:01.389154 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 295603us
I20260504 20:02:01.270937 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:01.262652 16035 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:01.372427 15973 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.444720 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:01.446935 13532 raft_consensus.cc:3060] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:01.449097 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:01.449900 13527 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:01.450243 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:01.450898 13526 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:01.451624 15605 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:01.453452 16034 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:01.444680 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:01.444108 12748 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 10.
I20260504 20:02:01.461985 13532 raft_consensus.cc:2468] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:01.275738 12646 catalog_manager.cc:5671] T 00b8418b3a8441ee845a0591a75a9f17 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 4 to 6, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 6 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260504 20:02:01.464903 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 4.62863e+06us
I20260504 20:02:01.437077 15943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57644 (request call id 3) took 18929 ms. Trace:
I20260504 20:02:01.462605 15698 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:01.479629 12790 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:02:01.482371 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 92472.1us
I20260504 20:02:01.480568 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:01.483906 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:02:01.478119 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800269bc0 after lost signal to thread 12741
I20260504 20:02:01.502437 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1800) took 5518 ms. Trace:
I20260504 20:02:01.502743 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.503186 12742 rpcz_store.cc:276] 0504 20:01:55.984013 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.984342 (+ 329us) service_pool.cc:224] Handling call
0504 20:01:55.984777 (+ 435us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:55.985051 (+ 274us) raft_consensus.cc:1534] Early marking committed up to index 4
0504 20:01:56.319923 (+334872us) log.cc:844] Serialized 14 byte log entry
0504 20:02:01.467678 (+5147755us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:01.480063 (+ 12385us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:02:01.484030 (+ 3967us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:01.493259 (+ 9229us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:01.502138 (+ 8879us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4648635,"prepare.queue_time_us":478481,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"thread_start_us":747446,"threads_started":3,"wal-append.queue_time_us":775976,"wal-append.run_cpu_time_us":23303,"wal-append.run_wall_time_us":2719393}
I20260504 20:02:01.470681 15943 rpcz_store.cc:276] 0504 20:01:42.472235 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:42.472384 (+ 149us) service_pool.cc:224] Handling call
0504 20:01:42.565193 (+ 92809us) log.cc:844] Serialized 487 byte log entry
0504 20:01:43.039336 (+474143us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b18000efaa0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598debe6 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598debe6 00007f01598deb80
0504 20:01:43.431121 (+391785us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b54006de820. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c92ef06 00007f015c92887a 00007f015c927af4 00007f015c90828a 00007f015c90f133 00007f015c90f0b9 00007f015c90f049 00007f015c90f011 00007f015c90e1dd 0000000000379254
0504 20:01:43.662748 (+231627us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b18000efaa0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598debe6 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b
0504 20:01:43.663422 (+ 674us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:02:01.389335 (+17725913us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17929,"raft.queue_time_us":39374,"raft.run_cpu_time_us":5857,"raft.run_wall_time_us":805822,"spinlock_wait_cycles":47475584,"thread_start_us":40742,"threads_started":1,"wal-append.queue_time_us":92080,"wal-append.run_cpu_time_us":18374,"wal-append.run_wall_time_us":2165272}
I20260504 20:02:01.489463 16080 raft_consensus.cc:2804] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:02:01.511086 12744 raft_consensus.cc:2468] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:01.508330 15831 consensus_queue.cc:1059] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:02:01.510874 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:01.503793 12819 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:02:01.506570 16030 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:01.515316 12748 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:01.510069 15982 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.518129 16080 raft_consensus.cc:697] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:01.522269 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.523283 12816 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:02:01.524355 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:01.525014 12822 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:01.524521 16080 consensus_queue.cc:237] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.530432 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:01.538071 12742 raft_consensus.cc:2393] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:01.539520 13527 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:02:01.547116 12646 catalog_manager.cc:5671] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 3 to 4, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2). New cstate: current_term: 4 leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:02:01.547127 12790 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:01.554129 12822 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 8.
I20260504 20:02:01.555362 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:01.527489 12693 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 0e8eba4dbb8d436f81062ca3bbf46788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 9)
I20260504 20:02:01.558027 16034 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:02:01.530362 15982 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.558697 16034 raft_consensus.cc:697] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:01.562273 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:01.563019 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:01.563527 13527 raft_consensus.cc:2393] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:01.578459 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:01.579452 12745 raft_consensus.cc:2393] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 3: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:01.587343 12742 raft_consensus.cc:2468] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
W20260504 20:02:01.588224 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 121288us
I20260504 20:02:01.589164 15982 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:01.592712 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:01.593714 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.594022 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.594847 12817 raft_consensus.cc:2393] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:01.595484 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.597328 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:01.596010 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.597268 12818 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:02:01.559924 16034 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.598970 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.587332 15973 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:01.601274 12744 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 6
I20260504 20:02:01.601346 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:01.602612 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1797) took 5623 ms. Trace:
I20260504 20:02:01.603206 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.606542 12824 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 9.
I20260504 20:02:01.595753 12819 raft_consensus.cc:2955] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:02:01.612953 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.613793 12816 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:02:01.608016 12820 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:01.618440 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.619967 12824 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
I20260504 20:02:01.620446 12821 raft_consensus.cc:1240] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
W20260504 20:02:01.624696 12865 connection.cc:254] RPC call timeout handler was delayed by -3.8898134779999509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1744s for a response
I20260504 20:02:01.628907 12822 raft_consensus.cc:2410] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260504 20:02:01.627539 12747 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:02:01.606315 12743 rpcz_store.cc:276] 0504 20:01:55.958982 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:55.959141 (+ 159us) service_pool.cc:224] Handling call
0504 20:01:55.959535 (+ 394us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:01:55.959779 (+ 244us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:55.959805 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:01:56.134441 (+174636us) spinlock_profiling.cc:243] Waited 6.87 ms on lock 0x7b2c00056c38. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015b2a8c81 00007f015b2b85e2 00007f015c9e9de1 00007f015c9ae81a 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22
0504 20:01:56.134920 (+ 479us) log.cc:844] Serialized 26 byte log entry
0504 20:01:56.310232 (+175312us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:01:56.866243 (+556011us) log.cc:844] Serialized 14 byte log entry
0504 20:02:01.538706 (+4672463us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:01.571292 (+ 32586us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:01.579257 (+ 7965us) raft_consensus.cc:1685] finished
0504 20:02:01.581704 (+ 2447us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:01.581920 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4520630,"prepare.queue_time_us":411152,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"spinlock_wait_cycles":15801728,"thread_start_us":644146,"threads_started":3,"wal-append.queue_time_us":248919,"wal-append.run_cpu_time_us":32625,"wal-append.run_wall_time_us":4423610}
I20260504 20:02:01.632431 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:01.647643 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.649307 12820 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 10.
I20260504 20:02:01.649375 12744 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:01.655815 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.659373 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.677160 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:01.679561 12820 raft_consensus.cc:2410] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260504 20:02:01.679575 12822 raft_consensus.cc:2376] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 8. Re-sending same reply.
I20260504 20:02:01.685523 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:01.661919 13526 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 9.
W20260504 20:02:01.700693 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 901821us
I20260504 20:02:01.703763 15975 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.704535 15975 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.706507 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:01.714267 12646 catalog_manager.cc:5671] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 2 to 3, leader changed from ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 3 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:02:01.715840 16011 raft_consensus.cc:3055] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:02:01.716360 16011 raft_consensus.cc:740] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:01.716204 12646 catalog_manager.cc:5671] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 3 to 4, leader changed from 732c3560a688494196d22a1a48f89968 (127.11.68.3) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 4 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:02:01.717269 16011 consensus_queue.cc:260] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.719772 16011 raft_consensus.cc:3060] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:01.723254 15827 consensus_queue.cc:1059] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260504 20:02:01.726188 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800274da0 after lost signal to thread 12750
I20260504 20:02:01.726624 12821 raft_consensus.cc:2468] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
W20260504 20:02:01.727067 12679 debug-util.cc:398] Leaking SignalData structure 0x7b080008b1c0 after lost signal to thread 12774
W20260504 20:02:01.727393 12679 debug-util.cc:398] Leaking SignalData structure 0x7b080008b1a0 after lost signal to thread 12775
W20260504 20:02:01.728647 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800091960 after lost signal to thread 12849
I20260504 20:02:01.728266 15986 consensus_queue.cc:1059] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260504 20:02:01.728951 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800091940 after lost signal to thread 12850
I20260504 20:02:01.729570 16058 raft_consensus.cc:3055] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:02:01.730108 16058 raft_consensus.cc:740] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:01.726455 16006 consensus_queue.cc:1059] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260504 20:02:01.739694 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800331f60 after lost signal to thread 12923
W20260504 20:02:01.744297 16107 log.cc:927] Time spent T 0e8eba4dbb8d436f81062ca3bbf46788 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260504 20:02:01.745165 15651 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:01.726222 15974 consensus_queue.cc:1059] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.020s
W20260504 20:02:01.749570 12679 debug-util.cc:398] Leaking SignalData structure 0x7b080033e960 after lost signal to thread 13557
W20260504 20:02:01.750674 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800343060 after lost signal to thread 15949
W20260504 20:02:01.751042 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800152460 after lost signal to thread 15951
W20260504 20:02:01.748871 12865 connection.cc:254] RPC call timeout handler was delayed by -2.6873170300000311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1545s for a response
W20260504 20:02:01.752784 12865 connection.cc:254] RPC call timeout handler was delayed by -2.2754560569999285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1966s for a response
W20260504 20:02:01.753201 12865 connection.cc:254] RPC call timeout handler was delayed by -2.2655412049998631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20260504 20:02:01.755734 12865 connection.cc:254] RPC call timeout handler was delayed by -1.9842373320000206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2335s for a response
W20260504 20:02:01.757194 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800263340 after lost signal to thread 15991
W20260504 20:02:01.757704 12679 debug-util.cc:398] Leaking SignalData structure 0x7b080027b760 after lost signal to thread 15993
I20260504 20:02:01.758174 13528 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
W20260504 20:02:01.760491 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800270900 after lost signal to thread 16010
I20260504 20:02:01.748955 15982 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:01.748052 15831 raft_consensus.cc:3055] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:02:01.748737 15958 raft_consensus.cc:3055] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:01.802883 12790 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:02:01.805203 13498 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:01.745790 15975 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:01.745894 15651 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.747299 16058 consensus_queue.cc:260] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:01.839185 12679 debug-util.cc:398] Leaking SignalData structure 0x7b080013eb80 after lost signal to thread 16031
I20260504 20:02:01.746966 15847 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:01.840418 12749 raft_consensus.cc:1240] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 4. Current term is 5. Ops: []
W20260504 20:02:01.859431 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800330bc0 after lost signal to thread 16048
W20260504 20:02:01.860023 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 103390us
W20260504 20:02:01.871636 12679 debug-util.cc:398] Leaking SignalData structure 0x7b08003a32e0 after lost signal to thread 16052
W20260504 20:02:01.872136 12679 debug-util.cc:398] Leaking SignalData structure 0x7b08000f7640 after lost signal to thread 16066
W20260504 20:02:01.872539 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800377d00 after lost signal to thread 16067
I20260504 20:02:01.839378 12646 catalog_manager.cc:5671] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 5 to 6, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 6 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:02:01.785614 16040 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:01.879038 12679 debug-util.cc:398] Leaking SignalData structure 0x7b0800152700 after lost signal to thread 16068
I20260504 20:02:01.842353 15993 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.842765 16034 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:02:01.810323 15831 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:01.839800 15951 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.790225 15386 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.810256 15958 raft_consensus.cc:740] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:01.840904 15974 raft_consensus.cc:3055] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:01.841782 15991 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:01.839769 16006 raft_consensus.cc:3055] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:02:01.837086 15982 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.790921 15949 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:01.907085 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1813) took 5011 ms. Trace:
I20260504 20:02:01.921846 12749 rpcz_store.cc:276] 0504 20:01:56.895766 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.897333 (+ 1567us) service_pool.cc:224] Handling call
0504 20:01:56.897773 (+ 440us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:01.878975 (+4981202us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:01.893580 (+ 14605us) spinlock_profiling.cc:243] Waited 4.94 s on lock 0x7b54002e59d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:01.907010 (+ 13430us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11366178688}
I20260504 20:02:01.906816 15974 raft_consensus.cc:740] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:02:01.923260 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:01.923748 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:01.923972 15974 consensus_queue.cc:260] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.925947 16031 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.926512 15983 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.926658 16031 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.927124 15983 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.929134 16031 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:01.929945 16049 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.914086 15958 consensus_queue.cc:260] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.930601 16049 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.931107 12750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1815) took 4944 ms. Trace:
I20260504 20:02:01.931478 15958 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:01.931720 12750 rpcz_store.cc:276] 0504 20:01:56.986631 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.986882 (+ 251us) service_pool.cc:224] Handling call
0504 20:01:56.987316 (+ 434us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:01:56.987617 (+ 301us) raft_consensus.cc:1534] Early marking committed up to index 2
0504 20:01:57.055368 (+ 67751us) log.cc:844] Serialized 14 byte log entry
0504 20:02:01.906809 (+4851441us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:01.930370 (+ 23561us) raft_consensus.cc:1645] Marking committed up to 2
0504 20:02:01.930490 (+ 120us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:01.930628 (+ 138us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:01.930833 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4572505,"prepare.queue_time_us":105426,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":9045,"thread_start_us":260135,"threads_started":2,"wal-append.queue_time_us":345489,"wal-append.run_cpu_time_us":1614,"wal-append.run_wall_time_us":1001657}
I20260504 20:02:01.934427 16070 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.934271 15831 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.935055 16070 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.895567 16040 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.942029 16019 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260504 20:02:01.929530 15983 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:01.943778 15991 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.946197 15991 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:01.946864 15991 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:01.947458 15991 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.949967 15949 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.950781 16010 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:01.917176 16006 raft_consensus.cc:740] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:02:01.954149 12714 connection.cc:254] RPC call timeout handler was delayed by -3.0958724469999197s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
I20260504 20:02:01.953752 16006 consensus_queue.cc:260] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:01.954636 12714 connection.cc:254] RPC call timeout handler was delayed by -3.0013724469998806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
W20260504 20:02:01.954999 12714 connection.cc:254] RPC call timeout handler was delayed by -2.880772446999913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
W20260504 20:02:01.955329 12714 connection.cc:254] RPC call timeout handler was delayed by -2.5567724469999575s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2316s for a response
I20260504 20:02:01.955634 16006 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:01.956360 16111 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.956146 16003 consensus_queue.cc:260] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.956930 16111 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.958020 15834 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:02:01.958287 12715 connection.cc:254] RPC call timeout handler was delayed by -2.9243860080000559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2464s for a response
I20260504 20:02:01.958734 15834 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.959213 12715 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:01.959648 16111 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:01.960242 16111 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:01.960752 16111 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:01.961176 15834 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:01.962577 16114 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260504 20:02:01.962826 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 330505us
I20260504 20:02:01.963088 16114 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:01.916968 15993 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:01.963581 16114 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:01.963958 12716 leader_election.cc:400] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 2. Current term is 3.
I20260504 20:02:01.964325 12716 leader_election.cc:403] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260504 20:02:01.965567 16010 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 2. Current term is 3.
W20260504 20:02:01.949991 16118 log.cc:927] Time spent T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
W20260504 20:02:01.963891 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:01.967159 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:01.967374 12716 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:02:01.955636 12714 connection.cc:254] RPC call timeout handler was delayed by -2.4235724469999695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2464s for a response
W20260504 20:02:01.969965 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 61928.6us
I20260504 20:02:01.941229 16070 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:01.925577 15974 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:01.968873 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:01.981720 12749 raft_consensus.cc:2393] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 8: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:02:01.982013 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 177915us
I20260504 20:02:01.970932 15834 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:02:01.957734 16003 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:01.983670 15827 consensus_queue.cc:1048] T 04d76f164cad4ed0bc8e39ce2403623f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260504 20:02:01.984349 12864 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:01.894567 16034 raft_consensus.cc:697] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:01.991212 16038 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:01.996968 15651 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:01.997184 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:01.996754 12646 catalog_manager.cc:5671] T 4472f9104c524735bcfeed8474933a75 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:02:01.997972 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:01.998771 12646 catalog_manager.cc:5671] T 6673c1682edf457ba633ba714a47587f P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:02:02.000907 15831 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:02.000629 12646 catalog_manager.cc:5671] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 4 to 5, leader changed from ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4) to 732c3560a688494196d22a1a48f89968 (127.11.68.3). New cstate: current_term: 5 leader_uuid: "732c3560a688494196d22a1a48f89968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } } }
I20260504 20:02:01.998773 12747 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:01.965493 15993 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:01.932781 16049 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:02.010113 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.010948 13524 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:02.011943 15982 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:01.971477 15986 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:01.895892 15847 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:02.014837 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 47710.6us
I20260504 20:02:01.949663 15991 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:01.908316 15951 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.040272 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.031433 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.040968 12864 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:02.031689 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.018134 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.032665 12893 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260504 20:02:02.017656 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.042903 13530 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:02:02.045116 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 557) took 4934 ms. Trace:
I20260504 20:02:02.047596 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:01.998809 16040 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:02.051978 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.017268 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.013784 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.047443 12744 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 6
I20260504 20:02:01.907262 15386 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.048882 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.018025 15834 raft_consensus.cc:697] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:02.047422 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.048024 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.066035 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.059026 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.065673 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.070398 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:02:02.070919 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:02.071429 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.071344 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.072166 12747 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:02.072357 15983 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:02.072876 15983 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.073241 15983 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:02.075248 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.053556 12743 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:02.100332 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.101246 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.102909 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.103758 12822 raft_consensus.cc:1240] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:02.104346 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.031940 15605 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 20:02:02.062625 15949 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:02.028999 16058 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:02.062940 13524 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:02.047385 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.079592 15386 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:02.084126 15983 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.001616 16070 consensus_queue.cc:1048] T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260504 20:02:02.090044 12750 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
W20260504 20:02:02.092211 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 44850.8us
I20260504 20:02:02.093365 16034 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.094406 15847 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:02.095080 15951 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:02.095345 16111 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:02.096443 12716 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:02.097280 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.098215 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.039891 15986 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.058387 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.106523 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.124135 12890 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:02.125182 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.125779 12890 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:02:02.126480 15983 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:02.126705 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.127887 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.128151 12892 raft_consensus.cc:2368] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 8. Current term is 9.
I20260504 20:02:02.092311 15834 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:02.106874 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.130576 12716 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:02.131803 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.106974 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:02:02.107721 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.116410 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.130616 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.123070 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.117331 12897 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:02:02.105783 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.184171 12750 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
W20260504 20:02:02.034545 12791 connection.cc:254] RPC call timeout handler was delayed by -4.8167208510000137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1636s for a response
W20260504 20:02:02.185214 12791 connection.cc:254] RPC call timeout handler was delayed by -4.3577208509999537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2146s for a response
I20260504 20:02:02.185287 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.120980 13497 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:02.114530 12715 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:02.106207 15605 raft_consensus.cc:697] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:02:02.174119 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.189743 12745 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:02.174717 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.179929 16114 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.190706 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:02:02.182371 12648 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:39338 (request call id 229) took 28767 ms (client timeout 15000 ms). Trace:
W20260504 20:02:02.185791 12791 connection.cc:254] RPC call timeout handler was delayed by -4.2848208510000632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
I20260504 20:02:02.186419 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.107558 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.202979 12646 catalog_manager.cc:5671] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 6 to 7, leader changed from 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2) to ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4). New cstate: current_term: 7 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:02:02.207746 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.208441 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.209481 12750 raft_consensus.cc:2393] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 8: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:02.045727 12895 rpcz_store.cc:276] 0504 20:01:57.103245 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:00.799583 (+3696338us) service_pool.cc:224] Handling call
0504 20:02:00.800145 (+ 562us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:00.800633 (+ 488us) raft_consensus.cc:1534] Early marking committed up to index 6
0504 20:02:00.816141 (+ 15508us) log.cc:844] Serialized 14 byte log entry
0504 20:02:01.971757 (+1155616us) log.cc:844] Serialized 14 byte log entry
0504 20:02:02.037371 (+ 65614us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:02.037453 (+ 82us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:02:02.037522 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:02.037626 (+ 104us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:02.037757 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":978639,"prepare.queue_time_us":173115,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":908688,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":2670,"spinlock_wait_cycles":3328,"thread_start_us":256482,"threads_started":3,"wal-append.queue_time_us":64773,"wal-append.run_cpu_time_us":2134,"wal-append.run_wall_time_us":1008511}
W20260504 20:02:02.209751 12791 connection.cc:254] RPC call timeout handler was delayed by -3.8168208509999886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2747s for a response
W20260504 20:02:02.211202 12791 connection.cc:254] RPC call timeout handler was delayed by -3.4709090639998976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
W20260504 20:02:02.211750 12791 connection.cc:254] RPC call timeout handler was delayed by -3.4709090639998976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
W20260504 20:02:02.211714 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 97260.6us
W20260504 20:02:02.212564 12791 connection.cc:254] RPC call timeout handler was delayed by -3.419920851000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31880000000000003s for a response
W20260504 20:02:02.213166 12791 connection.cc:254] RPC call timeout handler was delayed by -3.0020090639999353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
W20260504 20:02:02.213577 12791 connection.cc:254] RPC call timeout handler was delayed by -2.9813090639999018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2066s for a response
W20260504 20:02:02.214114 12791 connection.cc:254] RPC call timeout handler was delayed by -2.9291090639999311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21240000000000003s for a response
W20260504 20:02:02.214576 12791 connection.cc:254] RPC call timeout handler was delayed by -2.9291090639999311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21240000000000003s for a response
I20260504 20:02:02.215703 12742 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:02.217032 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:02:02.219810 12791 connection.cc:254] RPC call timeout handler was delayed by -2.9102090639999005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260504 20:02:02.220414 12791 connection.cc:254] RPC call timeout handler was delayed by -2.8454090639999094s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2217s for a response
W20260504 20:02:02.220875 12791 connection.cc:254] RPC call timeout handler was delayed by -2.4602090639999687s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2645s for a response
I20260504 20:02:02.221411 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.217927 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:02.223291 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.230662 16127 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:02.231015 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.223340 15847 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:02.233512 12742 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260504 20:02:02.234088 16114 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:02.235236 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1808) took 6155 ms. Trace:
I20260504 20:02:02.235616 12746 rpcz_store.cc:276] 0504 20:01:56.079904 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.080096 (+ 192us) service_pool.cc:224] Handling call
0504 20:01:56.080502 (+ 406us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:56.080892 (+ 390us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:56.080919 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:56.085947 (+ 5028us) log.cc:844] Serialized 52 byte log entry
0504 20:01:56.310756 (+224809us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:56.693112 (+382356us) log.cc:844] Serialized 14 byte log entry
0504 20:01:56.940727 (+247615us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b4800300258. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f01618e61a1 00007f01618dd237 00007f01618dd0ca 00007f01619d492f 00007f01619d45f7 00007f01619e8c1c 00007f01619e8bc8 00007f01619e8b48 00007f01619e8b03 00007f01619e7dac 00007f0160910b36
0504 20:02:01.888802 (+4948075us) log.cc:844] Serialized 14 byte log entry
0504 20:02:02.234630 (+345828us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:02.234751 (+ 121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:02.234874 (+ 123us) raft_consensus.cc:1685] finished
0504 20:02:02.234888 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:02.235067 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9873333,"prepare.queue_time_us":422595,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":108,"raft.queue_time_us":5264086,"raft.run_cpu_time_us":3541,"raft.run_wall_time_us":878800,"spinlock_wait_cycles":29781248,"thread_start_us":818098,"threads_started":4,"wal-append.queue_time_us":276812,"wal-append.run_cpu_time_us":49623,"wal-append.run_wall_time_us":4859260}
I20260504 20:02:02.239815 16017 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:02:02.240520 16017 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:02.241082 16017 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:02.197557 12821 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a"
dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:49576
I20260504 20:02:02.244042 12821 raft_consensus.cc:606] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:02.244540 12821 raft_consensus.cc:614] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260504 20:02:02.246105 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 115637us
I20260504 20:02:02.247442 12716 leader_election.cc:304] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:02.248883 15827 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:02.249629 15974 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:02.249754 15827 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.250613 15974 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.254091 15827 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:02.255623 12895 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:02:02.256592 12895 raft_consensus.cc:606] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:02.258888 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260504 20:02:02.261510 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260504 20:02:02.262889 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:02:02.263659 15986 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:02.264604 12714 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:02:02.191675 12648 rpcz_store.cc:269] 0504 20:01:33.414833 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:33.442974 (+ 28141us) service_pool.cc:224] Handling call
0504 20:02:02.182078 (+28739104us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:01:49.887131 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:01:49.916630 (+ 29499us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:01:49.916736 (+ 106us) write_op.cc:435] Acquired schema lock
0504 20:01:49.916769 (+ 33us) tablet.cc:662] Decoding operations
0504 20:01:49.928726 (+ 11957us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:01:49.928818 (+ 92us) write_op.cc:641] Partition lock acquired for write op
0504 20:01:49.928855 (+ 37us) tablet.cc:685] Acquiring locks for 14 operations
0504 20:01:49.951850 (+ 22995us) tablet.cc:701] Row locks acquired
0504 20:01:49.951908 (+ 58us) write_op.cc:260] PREPARE: finished
0504 20:01:49.952238 (+ 330us) write_op.cc:270] Start()
0504 20:01:49.952446 (+ 208us) write_op.cc:276] Timestamp: P: 1777924909952072 usec, L: 0
0504 20:01:49.952490 (+ 44us) op_driver.cc:348] REPLICATION: starting
0504 20:01:49.967260 (+ 14770us) log.cc:844] Serialized 5489 byte log entry
0504 20:01:51.124253 (+1156993us) op_driver.cc:464] REPLICATION: finished
0504 20:01:51.340433 (+216180us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f01598f382c 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f01603f0360
0504 20:01:51.496875 (+156442us) write_op.cc:301] APPLY: starting
0504 20:01:51.513265 (+ 16390us) tablet.cc:1367] starting BulkCheckPresence
0504 20:01:51.562581 (+ 49316us) tablet.cc:1370] finished BulkCheckPresence
0504 20:01:51.574065 (+ 11484us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:01:51.637505 (+ 63440us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:01:51.637922 (+ 417us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0504 20:01:51.637967 (+ 45us) write_op.cc:312] APPLY: finished
0504 20:01:51.638419 (+ 452us) log.cc:844] Serialized 101 byte log entry
0504 20:01:51.638616 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:51.638771 (+ 155us) write_op.cc:454] Released schema lock
0504 20:01:51.638835 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0504 20:01:51.736467 (+ 97632us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:01:51.738056 (+ 1589us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":15022,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":317852,"apply.run_cpu_time_us":5962,"apply.run_wall_time_us":218598,"num_ops":14,"prepare.queue_time_us":464081,"prepare.run_cpu_time_us":26907,"prepare.run_wall_time_us":349738,"raft.queue_time_us":383988,"raft.run_cpu_time_us":14038,"raft.run_wall_time_us":265153,"replication_time_us":1171564,"spinlock_wait_cycles":87686912,"thread_start_us":1089977,"threads_started":4,"wal-append.queue_time_us":396357,"wal-append.run_cpu_time_us":42800,"wal-append.run_wall_time_us":2270445}]]}
W20260504 20:02:02.270027 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 46329.1us
I20260504 20:02:02.270658 12820 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
W20260504 20:02:02.287624 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 181120us
I20260504 20:02:02.288918 12744 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:02.289856 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.291075 12646 catalog_manager.cc:5671] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260504 20:02:02.228804 12749 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:02:02.293644 12747 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260504 20:02:02.294811 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:02.217660 16038 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260504 20:02:02.273926 13498 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.279783 12890 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:02:02.301712 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:02:02.304901 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.253589 15974 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:02.300779 16038 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.305971 15698 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260504 20:02:02.323521 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.306578 16086 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:02.327183 12899 raft_consensus.cc:2410] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260504 20:02:02.317284 15951 raft_consensus.cc:2749] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:02:02.333251 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 85939us
I20260504 20:02:02.328459 13499 leader_election.cc:400] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 8. Current term is 9.
I20260504 20:02:02.336498 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:02.293406 16111 consensus_queue.cc:1059] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260504 20:02:02.337519 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.340690 12890 raft_consensus.cc:3060] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:02.338150 13499 leader_election.cc:403] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260504 20:02:02.339258 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 121965us
I20260504 20:02:02.339798 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.325347 16038 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:02.326155 15698 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:02:02.351126 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 237143us
I20260504 20:02:02.332449 15847 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:02.353266 12819 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:02.352430 13497 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:02.357753 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:02.359953 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.361313 12716 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:02.341511 16070 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:02.366905 12820 raft_consensus.cc:2393] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:02.335071 16086 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.352559 15698 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:02.347854 16108 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:02.372363 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.374229 12895 raft_consensus.cc:1275] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260504 20:02:02.374511 12899 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:02.358525 15834 raft_consensus.cc:2804] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:02:02.349100 15943 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260504 20:02:02.385433 12714 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.384249 16140 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:02.386137 16003 consensus_queue.cc:1059] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:02:02.329437 15605 consensus_queue.cc:237] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.390262 12791 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.403852 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.404637 12895 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:02.406088 12714 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:02:02.413096 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:02.412878 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 46845.3us
I20260504 20:02:02.415635 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.416725 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:02.417528 12821 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:02.360924 15847 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:02.422585 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.415707 15949 raft_consensus.cc:3055] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Stepping down as leader of term 8
I20260504 20:02:02.427107 12897 raft_consensus.cc:1275] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260504 20:02:02.425707 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 69273.5us
I20260504 20:02:02.425494 12741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1804) took 6351 ms. Trace:
I20260504 20:02:02.432917 12865 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:02.426394 16037 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:02:02.434098 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:02.434793 12892 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:02.423362 15834 raft_consensus.cc:697] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:02.437258 16086 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:02.437930 16086 raft_consensus.cc:2749] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:02.437704 15834 consensus_queue.cc:237] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.440946 15651 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260504 20:02:02.392871 16114 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:02.444916 12899 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
W20260504 20:02:02.445392 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 93079.7us
I20260504 20:02:02.446393 12819 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:02.447264 12821 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:02.410640 16111 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:02.371762 16070 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.453501 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.430121 12890 raft_consensus.cc:2468] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:02:02.453650 13497 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:02:02.428364 16080 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:02.463227 13497 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:02:02.430004 15949 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:02.427745 12741 rpcz_store.cc:276] 0504 20:01:56.020451 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:56.020681 (+ 230us) service_pool.cc:224] Handling call
0504 20:01:56.021023 (+ 342us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:01:56.031736 (+ 10713us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:01:56.031773 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:01:56.047243 (+ 15470us) log.cc:844] Serialized 52 byte log entry
0504 20:01:56.389427 (+342184us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b1800236fe0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f01598f3587 00007f01626ecbc9 00007f015990fcdb 00007f015990df7f 00007f0159910242 00007f015c5475ab 00007f015c54d0c3 00007f015c54d471 00007f015c969def
0504 20:01:57.099493 (+710066us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:01:57.165649 (+ 66156us) log.cc:844] Serialized 14 byte log entry
0504 20:02:02.036845 (+4871196us) log.cc:844] Serialized 14 byte log entry
0504 20:02:02.339844 (+302999us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:02.346858 (+ 7014us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:02.362681 (+ 15823us) raft_consensus.cc:1685] finished
0504 20:02:02.362704 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:02.371969 (+ 9265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4606641,"prepare.queue_time_us":108473,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":109,"raft.queue_time_us":4830889,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":17894,"spinlock_wait_cycles":9064832,"thread_start_us":1177330,"threads_started":4,"wal-append.queue_time_us":1549379,"wal-append.run_cpu_time_us":33585,"wal-append.run_wall_time_us":3379868}
I20260504 20:02:02.484476 16006 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:02.485181 16006 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.487293 16006 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:02.488291 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:02:02.490032 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 57263.6us
I20260504 20:02:02.491429 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.495293 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.495543 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.496059 12821 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:02:02.496659 15847 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:02.496814 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.497313 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.497789 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.498208 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.497745 15949 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.498797 12822 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:02.500823 12716 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:02.500913 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.502048 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.489549 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.502724 12750 raft_consensus.cc:1275] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260504 20:02:02.503633 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:02.461884 16111 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.509654 12818 raft_consensus.cc:2410] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:02:02.508525 12892 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:02:02.464833 16080 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.399214 16108 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.515830 12824 raft_consensus.cc:1275] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260504 20:02:02.515641 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 147166us
I20260504 20:02:02.516426 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.520386 12893 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:02:02.530459 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.530542 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.531495 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.530975 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:02.532070 12816 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:02.532315 13530 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 10.
I20260504 20:02:02.533283 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.533773 12715 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:02.533943 12820 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:02.536549 15834 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260504 20:02:02.537115 15834 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.536684 12825 raft_consensus.cc:2393] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 8: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:02.514075 16030 raft_consensus.cc:3055] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:02:02.538341 16030 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:02.538214 16038 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.539518 12744 raft_consensus.cc:2393] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:02.539290 16030 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.541018 16030 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:02.543062 16038 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:02.513559 16111 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:02.530503 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:02:02.554294 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 44166us
I20260504 20:02:02.523411 16080 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:02.583277 12817 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:02.575110 12714 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.578470 12895 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 9.
W20260504 20:02:02.573565 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.588328 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.582831 16114 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:02.590629 16114 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.590987 15943 raft_consensus.cc:2804] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:02:02.591440 15943 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:02.591856 15943 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:02.592861 16114 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:02.593412 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.594267 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:02.594913 13530 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:02.551079 15834 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:02.541046 16052 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:02.607609 12821 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 7. Current term is 8. Ops: []
W20260504 20:02:02.597718 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 134593us
W20260504 20:02:02.597695 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 92542.2us
I20260504 20:02:02.611002 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.611801 12817 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:02.613572 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.614303 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.550446 16037 raft_consensus.cc:697] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:02:02.620400 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.594563 16038 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:02.629271 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.630345 12821 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:02.632835 12817 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:02.641887 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.649019 16143 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:02:02.649695 16143 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:02.650164 16143 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:02.629422 16038 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.656926 12825 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:02:02.656476 12791 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.658062 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:02.640570 16137 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:02:02.674749 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:02:02.686043 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.662415 16038 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
W20260504 20:02:02.701978 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 55208.7us
W20260504 20:02:02.687456 15831 consensus_meta.cc:287] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.649s user 0.014s sys 0.000s
W20260504 20:02:02.708103 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 205745us
I20260504 20:02:02.711558 12865 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.673280 16137 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.726442 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:02:02.753998 16146 log.cc:927] Time spent T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20260504 20:02:02.762732 13530 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
I20260504 20:02:02.764703 12820 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 10.
I20260504 20:02:02.764986 12715 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:02.714346 16070 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:02.716290 16137 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:02.716715 16144 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:02.796995 16034 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:02:02.802114 16086 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:02:02.807320 16144 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.807897 16034 raft_consensus.cc:697] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:02.809147 16034 consensus_queue.cc:237] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:02.810316 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:02.787849 15698 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.813069 15698 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:02.773118 16052 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:02:02.813781 15698 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.018s
I20260504 20:02:02.831760 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.836687 12821 raft_consensus.cc:2393] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:02.837785 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.838558 12821 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:02:02.823252 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:02.840093 12825 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:02.823635 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.824462 16052 raft_consensus.cc:697] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:02.846632 16086 raft_consensus.cc:697] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:02.851295 16070 consensus_queue.cc:1059] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:02:02.849792 16086 consensus_queue.cc:237] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:02.827229 16017 consensus_meta.cc:287] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.585s user 0.008s sys 0.005s
I20260504 20:02:02.847301 12817 raft_consensus.cc:1240] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:02:02.854580 16017 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.852945 16070 raft_consensus.cc:3055] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:02.849777 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.831362 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:02.861305 12824 raft_consensus.cc:1240] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:02.848703 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.870589 12821 raft_consensus.cc:2410] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:02:02.851061 12863 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:02.861325 12865 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.882752 12714 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:02.884323 12825 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 9.
W20260504 20:02:02.893982 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 128142us
I20260504 20:02:02.848269 16052 consensus_queue.cc:237] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:02.826663 16144 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:02.881150 15993 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:02.894337 15831 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:02.894671 16006 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:02.903223 16108 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:02.904006 12714 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.860251 16070 raft_consensus.cc:740] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:02.906880 16070 consensus_queue.cc:260] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.908582 16070 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:02.910025 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 48800us
I20260504 20:02:02.910463 16006 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:02.910828 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.912601 12865 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.917097 13532 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:02.921769 16017 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:02.922830 16126 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260504 20:02:02.923760 16126 consensus_queue.cc:1059] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:02:02.924788 16126 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:02.925199 16126 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:02.926851 16127 raft_consensus.cc:3055] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Stepping down as leader of term 8
I20260504 20:02:02.927381 16127 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:02.930570 15974 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:02.931181 15974 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.931596 15974 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:02.933362 13527 raft_consensus.cc:1240] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:02.910876 16006 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:02.901952 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 50979.9us
I20260504 20:02:02.938293 15605 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:02:02.938509 12863 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:02.938881 15605 raft_consensus.cc:697] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:02.942898 12714 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:02.937672 16006 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:02.925652 16126 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:02.954640 15951 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:02.959676 15951 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.960111 15951 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:02.901386 15831 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:02.928553 16127 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.943085 16152 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:02.972230 13532 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260504 20:02:02.974267 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:02.786677 16037 consensus_queue.cc:237] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:02.904059 16108 consensus_queue.cc:1059] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
W20260504 20:02:02.961928 16058 consensus_meta.cc:287] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.847s user 0.006s sys 0.010s
I20260504 20:02:02.963423 15651 consensus_queue.cc:1059] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260504 20:02:02.971361 15831 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
W20260504 20:02:02.992624 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 302719us
I20260504 20:02:02.949829 16030 consensus_queue.cc:1059] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
W20260504 20:02:02.985469 16114 consensus_queue.cc:1175] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "732c3560a688494196d22a1a48f89968" responder_term: 8 status { last_received { term: 3 index: 3 } last_committed_idx: 2 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260504 20:02:03.002655 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:03.045499 12825 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260504 20:02:02.992906 16127 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:02.997026 13499 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:02.978163 16152 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:03.072819 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 75910.9us
I20260504 20:02:03.046512 16153 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:03.057017 16058 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:03.088860 13499 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:02.979990 15605 consensus_queue.cc:237] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.101671 12896 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:02:03.102418 15993 raft_consensus.cc:3055] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:03.102885 15993 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:03.103099 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1844) took 1005 ms. Trace:
I20260504 20:02:03.103102 12891 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:03.103700 12896 rpcz_store.cc:276] 0504 20:02:02.097752 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:02.107806 (+ 10054us) service_pool.cc:224] Handling call
0504 20:02:03.102903 (+995097us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b54002f67d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:03.103029 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1896207744}
I20260504 20:02:03.104406 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1841) took 1007 ms. Trace:
I20260504 20:02:03.104799 12891 rpcz_store.cc:276] 0504 20:02:02.096683 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:02.096945 (+ 262us) service_pool.cc:224] Handling call
0504 20:02:03.104238 (+1007293us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b540035c5d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:03.104346 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1379779968}
I20260504 20:02:03.107115 15834 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.110518 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:03.111299 12822 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Advancing to term 11
W20260504 20:02:03.113436 16080 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.517s user 0.007s sys 0.003s
I20260504 20:02:03.114398 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:03.124238 13528 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:03.124303 16080 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.126132 12819 raft_consensus.cc:2393] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:03.071774 16147 raft_consensus.cc:2749] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:02:03.127964 16080 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.116175 15974 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.121266 16153 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.130542 16153 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.132596 15974 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:03.122817 15993 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.134473 13528 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 11.
I20260504 20:02:03.135913 12822 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 11.
I20260504 20:02:03.136138 12715 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:02:03.136368 16140 consensus_meta.cc:287] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.733s user 0.009s sys 0.003s
I20260504 20:02:03.136859 16140 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 8. Current term is 9.
I20260504 20:02:03.139037 15949 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:03.141050 16152 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.144286 16038 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.146713 15951 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.148675 13524 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 9.
I20260504 20:02:03.149087 15951 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.109556 15834 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:03.157608 13497 leader_election.cc:336] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.950s (SENT)
I20260504 20:02:03.158422 13497 leader_election.cc:304] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:03.160100 16142 raft_consensus.cc:2749] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260504 20:02:03.047487 16143 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.162796 13526 raft_consensus.cc:1240] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:02:03.163949 16143 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.164788 16030 consensus_queue.cc:1048] T d083393cd9f34e0b9d827ec5efd931eb P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:02:03.166312 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:03.166594 16108 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 FOLLOWER]: Leader election won for term 11
I20260504 20:02:03.166841 16058 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:03.167168 12822 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:03.167235 16108 raft_consensus.cc:697] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:02:03.173878 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 85112.2us
I20260504 20:02:03.174012 16137 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.174818 16108 consensus_queue.cc:237] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.176290 16137 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:03.182091 13530 raft_consensus.cc:1240] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 9. Current term is 10. Ops: []
W20260504 20:02:03.184268 16111 consensus_meta.cc:287] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.637s user 0.013s sys 0.000s
I20260504 20:02:03.185058 12822 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:02:03.184804 16111 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.187049 12716 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:02:03.188215 13497 leader_election.cc:336] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 2.057s (SENT)
I20260504 20:02:03.188545 12714 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:03.189293 13499 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:03.195152 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:03.167341 16058 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:03.200616 16111 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.078673 15847 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20260504 20:02:03.197782 16143 consensus_queue.cc:1048] T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:02:03.190323 16017 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:03.198777 16158 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:02:03.198815 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:03.191959 16030 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:03.218009 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:03.202610 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:03.194231 15834 consensus_queue.cc:1059] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
W20260504 20:02:03.205320 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 449187us
I20260504 20:02:03.194185 15951 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election won for term 8
W20260504 20:02:03.195087 15943 consensus_meta.cc:287] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.603s user 0.008s sys 0.005s
I20260504 20:02:03.210456 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:03.210872 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:03.217058 12822 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:03.227314 12819 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:03.224680 12791 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:03.229156 16152 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:02:03.226718 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:03.232846 15834 raft_consensus.cc:3055] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Stepping down as leader of term 9
I20260504 20:02:03.233011 15951 raft_consensus.cc:697] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:03.233381 15834 raft_consensus.cc:740] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:03.234179 15951 consensus_queue.cc:237] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.234297 15834 consensus_queue.cc:260] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.235775 15834 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:03.229666 16152 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:03.237836 16152 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:03.232873 16144 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.008s
I20260504 20:02:03.242753 15943 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.245087 15943 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.246264 12716 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:03.222807 16006 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.247324 16030 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:02:03.247905 16030 raft_consensus.cc:697] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:03.247975 12716 leader_election.cc:400] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:02:03.248381 12716 leader_election.cc:403] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:02:03.248634 16006 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:03.249176 16114 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6.
I20260504 20:02:03.248920 16030 consensus_queue.cc:237] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.246186 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:03.254483 15847 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:02:03.254992 16144 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260504 20:02:03.207281 15993 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:03.256835 12791 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:03.259140 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 346622us
I20260504 20:02:03.270316 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:03.292640 12822 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
I20260504 20:02:03.230530 16038 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.294204 12714 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:03.314559 12819 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:02:03.317001 12714 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:02:03.333673 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 76998.4us
I20260504 20:02:03.273447 16144 raft_consensus.cc:2764] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260504 20:02:03.346798 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:03.325560 15834 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:03.329000 16052 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election won for term 9
W20260504 20:02:03.366955 12791 leader_election.cc:336] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Remote error: Timed out: Call waited in the queue past client deadline
W20260504 20:02:03.370518 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 111762us
W20260504 20:02:03.371045 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 431473us
W20260504 20:02:03.371232 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 107363us
I20260504 20:02:03.372839 12863 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:03.375097 12646 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.000s sys 0.031s
W20260504 20:02:03.375530 12646 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.543s user 0.000s sys 0.031s
I20260504 20:02:03.381033 12791 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:03.355540 15847 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:03.329262 16153 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260504 20:02:03.394181 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.366024 16140 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:03.364976 16038 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:03.394518 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.407773 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.370246 15651 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:03.416728 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.418267 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.412679 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.379158 16052 raft_consensus.cc:697] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:03.377983 15834 raft_consensus.cc:697] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:03.415330 12744 raft_consensus.cc:1275] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260504 20:02:03.414713 12750 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 9.
I20260504 20:02:03.376742 12864 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:03.417373 12749 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:03.429683 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.431730 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.434979 12742 raft_consensus.cc:2393] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:03.434603 15651 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.440023 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.441020 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.440936 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.371721 16126 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.442672 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.376235 15605 consensus_queue.cc:1059] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260504 20:02:03.440519 16086 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:02:03.444229 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 91578.6us
I20260504 20:02:03.444806 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.445204 15834 consensus_queue.cc:237] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.449949 16038 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.450371 16052 consensus_queue.cc:237] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.451092 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.452521 12747 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 9.
I20260504 20:02:03.414203 16144 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:03.411078 16140 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:03.455029 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 74957.7us
I20260504 20:02:03.455310 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:02:03.454481 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 77844us
I20260504 20:02:03.459535 12746 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:03.462865 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.395262 15847 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.485335 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:02:03.487332 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 64199.1us
W20260504 20:02:03.496583 12791 leader_election.cc:336] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Remote error: Timed out: Call waited in the queue past client deadline
I20260504 20:02:03.505705 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:03.506485 16126 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.507109 15651 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.507829 12791 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:03.508535 16140 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:03.509131 15847 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.470948 16152 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.514858 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:03.515767 12822 raft_consensus.cc:2393] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:03.520306 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:03.521154 12749 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:02:03.523154 13499 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:03.523514 12748 raft_consensus.cc:1240] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:03.478695 16058 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.520306 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:03.542744 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:02:03.511756 16144 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:02:03.552627 16160 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:03.532425 16168 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:02:03.557065 16058 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:03.559898 13524 raft_consensus.cc:1240] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:02:03.562043 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:03.569101 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.580430 12822 raft_consensus.cc:2393] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:02:03.590209 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 99748.3us
I20260504 20:02:03.591593 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.591948 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.592491 12895 raft_consensus.cc:2393] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:03.591300 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.592945 12891 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:02:03.591377 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:03.595760 12896 raft_consensus.cc:2393] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:03.597445 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.599416 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:03.600227 12892 raft_consensus.cc:3060] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:03.601234 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.602083 12893 raft_consensus.cc:2368] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7.
I20260504 20:02:03.606650 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.608150 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57934 (request call id 501) took 1108 ms. Trace:
I20260504 20:02:03.608642 12747 rpcz_store.cc:276] 0504 20:02:02.488003 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:02.496540 (+ 8537us) service_pool.cc:224] Handling call
0504 20:02:03.595785 (+1099245us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b540022fbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:03.595919 (+ 134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196300032}
I20260504 20:02:03.578512 16126 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:03.570477 16162 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:03.579298 15847 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:03.578215 16160 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:03.593549 16185 raft_consensus.cc:3055] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:02:03.628131 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.632198 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.604802 16168 raft_consensus.cc:697] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [term 9 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:03.563755 15831 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.632295 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.638793 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.610361 15949 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:03.644220 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:03.658285 12816 raft_consensus.cc:1813] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Handling vote request from an unknown peer 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:03.589146 16006 consensus_queue.cc:1059] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260504 20:02:03.607797 15605 consensus_queue.cc:1048] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260504 20:02:03.669716 12816 raft_consensus.cc:2393] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:02:03.656599 12892 raft_consensus.cc:2468] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
W20260504 20:02:03.679906 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 76796.9us
I20260504 20:02:03.684496 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.640900 16160 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:03.642587 16185 raft_consensus.cc:740] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:03.688413 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:03.634440 16126 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.695863 16184 consensus_queue.cc:1059] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260504 20:02:03.697057 16058 consensus_queue.cc:1059] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:02:03.733078 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:03.648710 16108 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:03.689179 12890 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 9: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
W20260504 20:02:03.748421 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 99613.1us
I20260504 20:02:03.749907 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.763389 12892 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:02:03.638233 16162 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.681991 16034 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:02:03.752782 16156 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260504 20:02:03.765676 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.641285 15847 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.772148 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:03.670693 15949 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.782709 12715 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:03.792444 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:03.800971 13524 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260504 20:02:03.808948 16070 raft_consensus.cc:3055] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
W20260504 20:02:03.789499 16175 log.cc:927] Time spent T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.297s user 0.000s sys 0.001s
I20260504 20:02:03.771818 16038 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:03.862197 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:03.862988 12895 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:03.865463 12819 raft_consensus.cc:1275] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260504 20:02:03.866056 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 64989.6us
I20260504 20:02:03.865087 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:03.868002 12818 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:03.865722 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:03.865032 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:03.870378 12820 raft_consensus.cc:1275] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260504 20:02:03.780668 16034 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:03.769266 16108 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.809587 16070 raft_consensus.cc:740] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:03.870792 16152 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:03.892392 12823 raft_consensus.cc:1240] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260504 20:02:03.901154 12822 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 7: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:02:03.895836 16176 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.383s user 0.002s sys 0.000s
I20260504 20:02:03.900305 16003 raft_consensus.cc:3055] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Stepping down as leader of term 4
W20260504 20:02:04.063136 13530 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.053s sys 0.115s
I20260504 20:02:03.891594 12816 raft_consensus.cc:2393] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:04.085242 16003 raft_consensus.cc:740] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:02:04.090029 13530 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s user 0.053s sys 0.116s
W20260504 20:02:04.090206 15993 consensus_meta.cc:287] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.828s user 0.014s sys 0.014s
I20260504 20:02:04.095368 12743 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:04.070678 12895 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:02:03.870829 15605 raft_consensus.cc:3055] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
W20260504 20:02:03.866523 16157 log.cc:927] Time spent T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
W20260504 20:02:03.951586 16173 log.cc:927] Time spent T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.263s user 0.001s sys 0.000s
I20260504 20:02:04.076011 16198 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.078331 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:04.079205 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:04.124406 12891 raft_consensus.cc:2393] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:03.791349 16168 consensus_queue.cc:237] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:04.121201 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:04.067261 16038 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:03.905138 16114 raft_consensus.cc:3060] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:04.131587 12897 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:02:03.901906 16181 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.200s
I20260504 20:02:04.099771 15605 raft_consensus.cc:740] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:04.136884 12896 raft_consensus.cc:3055] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:02:04.137048 16201 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.138267 16189 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.138469 16198 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.151044 16006 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.151597 16006 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.152402 16034 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:04.152825 16198 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:04.139285 16192 log.cc:927] Time spent T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.345s user 0.001s sys 0.000s
I20260504 20:02:04.146238 12896 raft_consensus.cc:740] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:04.149072 15605 consensus_queue.cc:260] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.150171 16201 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:03.810755 16185 consensus_queue.cc:260] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:04.096131 15831 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:04.149642 16172 log.cc:927] Time spent T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.485s user 0.000s sys 0.001s
W20260504 20:02:04.141229 16177 log.cc:927] Time spent T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.493s user 0.000s sys 0.002s
I20260504 20:02:04.224499 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:04.153666 16006 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:04.221724 12896 consensus_queue.cc:260] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:04.233058 12896 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:04.155215 16108 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:04.225787 16126 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:04.251391 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:04.252475 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:04.257676 13530 raft_consensus.cc:2393] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:02:04.260574 12894 raft_consensus.cc:2393] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:02:04.250119 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 473690us
W20260504 20:02:04.241992 15943 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.000s sys 0.008s
W20260504 20:02:04.272158 12791 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.762s user 0.030s sys 0.056s
W20260504 20:02:04.274837 12791 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.765s user 0.031s sys 0.057s
W20260504 20:02:04.275223 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 765684us
I20260504 20:02:04.151088 16189 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.275800 12896 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260504 20:02:04.279639 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:04.143635 16070 consensus_queue.cc:260] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:04.282974 13524 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:02:04.291555 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:04.291867 12715 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:04.292310 12743 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 9.
I20260504 20:02:04.283366 12896 pending_rounds.cc:85] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: Aborting all ops after (but not including) 5
I20260504 20:02:04.224354 16038 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:04.294939 12896 pending_rounds.cc:107] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968: Aborting uncommitted NO_OP operation due to leader change: 3.6
W20260504 20:02:04.274034 15943 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.000s sys 0.009s
I20260504 20:02:04.294755 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:04.295377 12896 raft_consensus.cc:2889] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:02:04.248483 15949 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:04.260058 15847 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:04.300005 12898 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:02:04.314245 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:04.306288 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:04.315349 13526 raft_consensus.cc:2393] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 6: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:02:04.315485 12823 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 9: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:04.315629 16186 consensus_queue.cc:1059] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260504 20:02:04.316470 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:04.317909 16179 raft_consensus.cc:993] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968: : Instructing follower 82787d8e64d24ca885ca517aa8ed1be2 to start an election
I20260504 20:02:04.318413 12894 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:04.318419 16179 raft_consensus.cc:1081] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Signalling peer 82787d8e64d24ca885ca517aa8ed1be2 to start an election
I20260504 20:02:04.318437 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:04.320843 12895 raft_consensus.cc:2393] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:04.316563 16186 raft_consensus.cc:3055] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 20:02:04.321751 16186 raft_consensus.cc:740] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:04.322608 16186 consensus_queue.cc:260] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.298628 16162 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:04.323987 16186 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:04.307678 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:04.307261 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:04.309636 16204 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.310515 16197 consensus_queue.cc:1048] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260504 20:02:04.311142 16006 consensus_queue.cc:1048] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260504 20:02:04.311482 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:04.356467 12744 raft_consensus.cc:1275] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260504 20:02:04.347832 12892 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:02:04.297254 15651 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:04.317673 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:02:04.308382 16199 log.cc:927] Time spent T 5409cb132bc14d1cafaa29b22ced9dc9 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20260504 20:02:04.360421 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:04.335563 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:04.337357 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:04.361196 12746 raft_consensus.cc:3060] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 7
I20260504 20:02:04.361455 12750 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:04.281878 15831 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.239264 16003 consensus_queue.cc:260] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.358153 12892 raft_consensus.cc:606] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:04.327369 12791 leader_election.cc:336] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Remote error: Timed out: Call waited in the queue past client deadline
I20260504 20:02:04.375000 12791 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:04.326119 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:04.362126 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:04.309576 16038 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:04.359305 12820 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:04.357925 12891 raft_consensus.cc:2393] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:04.364769 12743 raft_consensus.cc:1240] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:02:04.381542 12716 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:04.382164 12741 raft_consensus.cc:2410] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260504 20:02:04.381889 12863 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:04.394299 12820 raft_consensus.cc:1240] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:04.395989 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:04.397141 12823 raft_consensus.cc:2393] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:04.360512 15651 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:04.397974 15651 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:04.398286 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:04.400431 12816 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260504 20:02:04.404534 16214 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:04.405329 16214 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:04.405776 16214 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:04.377561 15831 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.380432 12892 raft_consensus.cc:614] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260504 20:02:04.408607 12818 consensus_meta.cc:287] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.537s user 0.011s sys 0.001s
I20260504 20:02:04.410323 12818 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:04.411821 12863 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:04.412575 12897 raft_consensus.cc:1240] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:02:04.413903 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:04.415455 12819 raft_consensus.cc:1240] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:02:04.415812 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:04.416908 12825 raft_consensus.cc:2393] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:04.418711 12746 raft_consensus.cc:2468] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:04.419656 15831 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:04.415026 13499 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:04.438800 16201 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:04.420981 16006 raft_consensus.cc:2804] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:04.450659 12824 raft_consensus.cc:1275] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260504 20:02:04.408574 16003 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:04.392900 16070 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:04.392156 16153 raft_consensus.cc:2749] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260504 20:02:04.464600 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:04.465540 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:04.473249 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 174072us
W20260504 20:02:04.480949 12862 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:04.492789 12791 leader_election.cc:336] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151): Remote error: Timed out: Call waited in the queue past client deadline
I20260504 20:02:04.499645 12791 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:04.511257 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:04.575024 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 63946us
I20260504 20:02:04.576259 12865 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:04.609961 13524 raft_consensus.cc:1240] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 3. Current term is 4. Ops: []
I20260504 20:02:04.610729 12865 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:04.488276 16006 raft_consensus.cc:697] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:04.618741 16189 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:04.614164 15831 raft_consensus.cc:2749] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:02:04.621865 16086 consensus_meta.cc:287] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.157s user 0.015s sys 0.001s
I20260504 20:02:04.647233 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:02:04.710316 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 222039us
W20260504 20:02:04.723475 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 112822us
I20260504 20:02:04.727327 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:04.836359 12824 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:04.836380 12865 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:04.728293 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:04.730847 12817 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
W20260504 20:02:04.736214 16218 log.cc:927] Time spent T dfa79aedaea6499facdfc14a3e7467f8 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.248s user 0.000s sys 0.001s
I20260504 20:02:04.740199 16006 consensus_queue.cc:237] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.746263 12898 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:02:04.842585 12898 raft_consensus.cc:606] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:04.843142 12898 raft_consensus.cc:614] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:02:04.634121 16168 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:04.766764 16222 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.767359 15831 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.781921 16209 raft_consensus.cc:2764] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260504 20:02:04.565938 16204 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.853875 12822 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:04.728307 15651 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:04.712481 16086 raft_consensus.cc:2749] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:02:04.805665 16160 consensus_meta.cc:287] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.043s user 0.015s sys 0.000s
I20260504 20:02:04.860183 12749 raft_consensus.cc:1275] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:02:04.615186 16184 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:02:04.614647 16153 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 20:02:04.839686 12865 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:02:04.849342 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 114623us
W20260504 20:02:04.821345 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 205785us
W20260504 20:02:04.845142 16114 consensus_meta.cc:287] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.687s user 0.011s sys 0.002s
I20260504 20:02:04.870241 12899 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:02:04.856554 16222 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.872977 16214 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.875424 15651 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:04.873795 16222 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:04.874698 12865 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:04.866596 16137 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:04.868407 16114 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:02:04.895514 12865 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:04.856817 15831 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:04.900723 16137 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:04.909942 16223 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:02:04.910831 16223 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:04.911303 16223 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:04.915066 16214 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:04.885666 16184 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:04.890199 16222 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:02:04.921211 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:04.921442 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 55403.2us
I20260504 20:02:04.924180 16204 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:04.866513 16160 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:04.929639 16238 raft_consensus.cc:2749] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:04.930233 16160 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:04.931371 12863 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:04.925763 16137 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:04.932245 16238 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:02:04.932682 16238 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:04.933044 16238 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:04.934104 12863 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:04.943329 12863 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:02:04.943310 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 61630.7us
I20260504 20:02:04.950964 12890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2074) took 1380 ms. Trace:
I20260504 20:02:04.944382 16203 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:04.951648 12890 rpcz_store.cc:276] 0504 20:02:03.568498 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:03.568676 (+ 178us) service_pool.cc:224] Handling call
0504 20:02:04.948214 (+1379538us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54001fb3d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:04.948414 (+ 200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":249788672}
I20260504 20:02:04.958423 13528 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260504 20:02:04.959837 16236 consensus_queue.cc:1059] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260504 20:02:04.961205 16236 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260504 20:02:04.965952 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:04.952136 16006 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:04.963038 16203 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:04.990541 16234 log.cc:927] Time spent T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260504 20:02:04.995272 16203 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:04.987556 16006 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:04.997189 16006 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:02:05.009083 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 113726us
W20260504 20:02:05.009716 16195 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.878s user 0.004s sys 0.003s
I20260504 20:02:05.010195 12865 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:05.011842 16186 consensus_meta.cc:287] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.668s user 0.008s sys 0.008s
I20260504 20:02:05.018291 16222 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:05.019876 12791 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.021498 16152 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:02:05.022182 16152 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:05.021458 16238 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.022578 16152 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.024492 16238 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:05.028821 12791 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.033612 16153 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:02:05.035384 16217 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:05.036233 16217 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.037917 12791 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:05.038556 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:05.039355 12865 leader_election.cc:304] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:02:05.030769 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:05.031659 16152 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.044699 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.045256 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.045430 12818 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:05.045934 12825 raft_consensus.cc:2468] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:02:05.044726 16201 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:05.047260 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.047922 12817 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:05.050215 15993 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:05.050566 16205 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:05.050763 15993 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.051280 16217 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:05.051192 16205 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.051957 12819 raft_consensus.cc:1275] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260504 20:02:05.052887 15993 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:05.051232 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.057169 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:05.058944 13528 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:05.060546 12863 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:05.067978 13530 raft_consensus.cc:1275] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260504 20:02:05.069453 12791 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.074537 12818 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:05.075569 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.075878 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.076565 12816 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.078375 12865 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.053339 16205 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:05.082329 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:05.083994 12865 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.081732 13528 raft_consensus.cc:2468] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
I20260504 20:02:05.077227 16162 raft_consensus.cc:2764] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260504 20:02:05.096836 16171 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:05.085055 16170 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:05.098371 16171 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:05.099115 16236 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:05.100855 16171 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:05.086196 12865 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.104341 12865 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.104651 12863 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:05.061574 16201 raft_consensus.cc:2749] T e44b757a449142a38ed04025e01cb4a5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:05.106132 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.069038 16197 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:02:05.109467 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.111215 16197 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:05.082523 16186 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:05.111644 16197 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.111726 16152 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:05.112138 16241 raft_consensus.cc:2749] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260504 20:02:05.112749 16153 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:05.112270 16186 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:02:05.113785 16186 raft_consensus.cc:697] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:05.103812 16198 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:05.084897 16235 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260504 20:02:05.114760 16186 consensus_queue.cc:237] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.110134 12865 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.119995 12791 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.121790 12895 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:02:05.122623 12895 raft_consensus.cc:606] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.123128 12895 raft_consensus.cc:614] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260504 20:02:05.124811 16197 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.127573 16197 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:05.085402 16185 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.102475 15831 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:05.138608 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.139397 12818 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.139786 16238 consensus_queue.cc:1059] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:02:05.141403 16238 raft_consensus.cc:3055] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Stepping down as leader of term 8
I20260504 20:02:05.141959 16238 raft_consensus.cc:740] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:05.144106 16235 raft_consensus.cc:2749] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260504 20:02:05.138439 12865 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:05.144901 12823 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260504 20:02:05.147265 12791 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.148576 12895 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:02:05.149245 12895 raft_consensus.cc:606] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Received request to transfer leadership to TS 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.143373 16238 consensus_queue.cc:260] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.151963 16238 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:05.082859 16187 consensus_queue.cc:1048] T 593a993b4ff341c9b98629e2c051ce00 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260504 20:02:05.160651 12865 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.163709 12818 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:05.164149 12865 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:05.165578 12863 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:05.167389 12791 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.135010 16242 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260504 20:02:05.169997 12863 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:05.168486 16201 raft_consensus.cc:2749] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260504 20:02:05.173913 16170 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:02:05.174067 16153 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:02:05.174504 16170 raft_consensus.cc:697] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:05.175523 16170 consensus_queue.cc:237] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.176533 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae27dbfd0ab84c9d8c95896c0c4cf502" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:05.176584 12791 leader_election.cc:304] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.178112 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.179706 16152 raft_consensus.cc:2749] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:05.180370 16152 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:02:05.182272 16203 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.184392 16203 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:05.166581 16235 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:05.186231 16235 raft_consensus.cc:697] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:05.186231 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.187378 12819 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:05.187315 16235 consensus_queue.cc:237] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.191455 16006 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.192466 12791 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.193184 16153 consensus_queue.cc:1048] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260504 20:02:05.196287 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.168584 16198 consensus_queue.cc:1048] T 6d6a540765d9427aa634550381fa5eb2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260504 20:02:05.158671 16179 raft_consensus.cc:2764] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260504 20:02:05.193914 16152 consensus_queue.cc:1048] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260504 20:02:05.202025 12895 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2"
dest_uuid: "732c3560a688494196d22a1a48f89968"
mode: GRACEFUL
new_leader_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
from {username='slave'} at 127.0.0.1:58902
I20260504 20:02:05.212226 12819 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:05.216069 12895 raft_consensus.cc:606] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Received request to transfer leadership to TS 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:05.219019 12863 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:05.200749 16186 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260504 20:02:05.200502 16006 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:05.196086 16223 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.235103 16200 raft_consensus.cc:3055] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Stepping down as leader of term 3
W20260504 20:02:05.242448 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 50074.9us
I20260504 20:02:05.222038 16188 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:02:05.245131 12791 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.250533 12816 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:05.252245 16034 raft_consensus.cc:2804] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:02:05.252991 16034 raft_consensus.cc:697] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:05.253898 16188 raft_consensus.cc:697] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:05.254061 16153 raft_consensus.cc:2764] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260504 20:02:05.254951 16188 consensus_queue.cc:237] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.244285 16200 raft_consensus.cc:740] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:05.256031 12791 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.256923 16152 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:05.257473 16152 raft_consensus.cc:697] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:05.257004 16200 consensus_queue.cc:260] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:05.258423 16152 consensus_queue.cc:237] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:05.259588 16244 log.cc:927] Time spent T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260504 20:02:05.262115 12863 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:05.266988 16198 raft_consensus.cc:2804] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:02:05.268149 12791 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.268757 16034 consensus_queue.cc:237] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.275821 16223 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:05.274209 16198 raft_consensus.cc:697] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:05.276935 16198 consensus_queue.cc:237] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.283962 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.261637 16200 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:05.306105 16162 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.306918 16162 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.299455 16250 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:02:05.311178 16250 raft_consensus.cc:697] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:05.312191 16250 consensus_queue.cc:237] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:05.337518 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 69507.1us
I20260504 20:02:05.342269 12791 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:05.378849 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 309124us
I20260504 20:02:05.395011 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.395720 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.395980 12748 raft_consensus.cc:3060] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.398936 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 512) took 1666 ms. Trace:
I20260504 20:02:05.397356 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.403728 12745 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 8
I20260504 20:02:05.405352 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.399528 12747 rpcz_store.cc:276] 0504 20:02:03.723207 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:03.727570 (+ 4363us) service_pool.cc:224] Handling call
0504 20:02:03.752403 (+ 24833us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:02:04.066210 (+313807us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:04.066250 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:02:04.079668 (+ 13418us) log.cc:844] Serialized 52 byte log entry
0504 20:02:04.343710 (+264042us) raft_consensus.cc:1645] Marking committed up to 8
0504 20:02:04.360774 (+ 17064us) log.cc:844] Serialized 14 byte log entry
0504 20:02:05.025155 (+664381us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b180045d060. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f0162777251 00007f0162783ab1 00007f0159884e1a 00007f0162f265c9 00007f01597431f0 00007f015990db24 00007f015990da95 00007f015c9a6440 00007f015c9b8af6 00007f015c9c9667 00007f015c9c95e8
0504 20:02:05.025697 (+ 542us) log.cc:844] Serialized 14 byte log entry
0504 20:02:05.375017 (+349320us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:05.375150 (+ 133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:05.389300 (+ 14150us) raft_consensus.cc:1685] finished
0504 20:02:05.389340 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:05.389577 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":366660,"prepare.queue_time_us":15071,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":93,"raft.queue_time_us":1041968,"raft.run_cpu_time_us":502,"raft.run_wall_time_us":59931,"spinlock_wait_cycles":444876032,"thread_start_us":420153,"threads_started":2,"wal-append.queue_time_us":498443}
W20260504 20:02:05.408586 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 89629.5us
I20260504 20:02:05.376436 16204 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:02:05.374923 16215 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.410581 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:05.413684 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.420337 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.420619 12750 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 8
I20260504 20:02:05.421243 12747 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.417162 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.418346 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.422422 12741 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:05.423808 12743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.427872 12816 raft_consensus.cc:1275] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260504 20:02:05.410051 12746 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0"
dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
from {username='slave'} at 127.0.0.1:57462
I20260504 20:02:05.433601 12746 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260504 20:02:05.434137 12746 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.434223 12742 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:02:05.408843 16169 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.438318 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.439226 12741 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:05.441864 12715 leader_election.cc:304] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:02:05.444419 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 1.0672e+06us
I20260504 20:02:05.390197 16003 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260504 20:02:05.423929 16034 raft_consensus.cc:2764] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260504 20:02:05.454429 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.458488 12744 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 8
I20260504 20:02:05.427539 16215 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:05.466881 12745 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:05.468711 12750 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:05.467113 12716 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:05.435990 16182 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.476125 12747 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:05.480022 12746 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.481699 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.453945 16235 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.454941 16198 consensus_queue.cc:1048] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260504 20:02:05.484244 16235 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.454218 16255 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.486178 16255 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.486717 16235 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:05.488397 16255 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:05.493744 16215 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:05.473632 16038 raft_consensus.cc:2804] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election won for term 9
W20260504 20:02:05.496371 16247 log.cc:927] Time spent T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.272s user 0.000s sys 0.002s
W20260504 20:02:05.498171 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 52951.7us
I20260504 20:02:05.498520 16162 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:05.502367 11536 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260504 20:02:05.502576 16171 consensus_queue.cc:1059] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260504 20:02:05.442253 15651 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:05.503927 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1859) took 1175 ms. Trace:
I20260504 20:02:05.502374 12716 leader_election.cc:400] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7.
I20260504 20:02:05.504647 12716 leader_election.cc:403] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:02:05.483964 12745 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:05.507905 16003 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:02:05.508745 16003 raft_consensus.cc:3055] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Stepping down as leader of term 8
I20260504 20:02:05.508194 15834 consensus_queue.cc:1048] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:02:05.509368 16003 raft_consensus.cc:740] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:05.504374 12749 rpcz_store.cc:276] 0504 20:02:04.328560 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:04.329927 (+ 1367us) service_pool.cc:224] Handling call
0504 20:02:04.359940 (+ 30013us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:04.860887 (+500947us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:05.503658 (+642771us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b540007c6d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:05.503851 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134596992}
W20260504 20:02:05.514945 12899 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2082) took 1890 ms (client timeout 2188 ms). Trace:
W20260504 20:02:05.515561 12899 rpcz_store.cc:269] 0504 20:02:03.624244 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:03.632378 (+ 8134us) service_pool.cc:224] Handling call
0504 20:02:05.514633 (+1882255us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540013b5d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:05.514815 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2794175872}
W20260504 20:02:05.516793 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 49303.6us
I20260504 20:02:05.510305 16003 consensus_queue.cc:260] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.518972 12714 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:05.526409 12747 raft_consensus.cc:1240] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:05.529320 12744 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:05.533245 12744 raft_consensus.cc:1275] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260504 20:02:05.534778 12716 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:02:05.535086 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 192971us
I20260504 20:02:05.536098 12716 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:05.536134 12791 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:05.537176 12716 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.538389 12746 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:05.539817 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:05.447172 16169 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.547118 12716 leader_election.cc:304] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.506464 16171 consensus_queue.cc:1048] T 7a80bf1f95a74ce09071a550fe15996f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260504 20:02:05.515719 16038 raft_consensus.cc:697] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:05.551681 12716 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.547454 12865 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:05.554709 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.561158 12820 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260504 20:02:05.556829 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:05.556545 12816 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:05.562994 12865 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:05.565732 16169 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:05.566774 16187 consensus_queue.cc:1048] T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:02:05.571885 12745 raft_consensus.cc:2468] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
W20260504 20:02:05.572357 13497 leader_election.cc:336] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415): Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.913s (SENT)
I20260504 20:02:05.572842 13497 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:05.574719 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.574977 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.575594 12745 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:05.534176 16198 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.511873 15651 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:02:05.577427 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:05.579092 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.594022 12818 raft_consensus.cc:2368] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8.
I20260504 20:02:05.571219 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.543985 16114 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 6. Current term is 7.
I20260504 20:02:05.591614 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.506554 16182 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.546255 16111 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:05.550556 16215 raft_consensus.cc:2804] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:05.614760 16038 consensus_queue.cc:237] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:05.622442 16215 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:05.623018 16215 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:05.624198 16252 log.cc:927] Time spent T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.265s user 0.003s sys 0.000s
I20260504 20:02:05.546062 15993 raft_consensus.cc:2749] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 20:02:05.609051 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.626736 16070 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.550365 16204 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:05.627512 16070 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:05.603735 16153 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:02:05.628916 16153 raft_consensus.cc:697] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:05.557744 16205 raft_consensus.cc:2749] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:05.576978 15834 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.051s
W20260504 20:02:05.616820 12696 auto_rebalancer.cc:285] failed to send replica move request: Illegal state: Replica 5ff0ef72ec3b4667893a0420d3c7f923 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:02:05.581382 16140 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260504 20:02:05.637317 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 43885.3us
W20260504 20:02:05.644883 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 108871us
W20260504 20:02:05.646858 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:05.645599 12741 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 10.
I20260504 20:02:05.650709 12716 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:05.652295 12716 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:05.576157 16006 raft_consensus.cc:2804] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:02:05.663707 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.577165 16127 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:05.664498 12820 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:05.579993 16222 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:02:05.616060 15651 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:05.559302 16003 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:05.613513 16189 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:05.697242 12714 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.626765 16182 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:05.676188 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.665400 12865 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:02:05.713079 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 47814.5us
I20260504 20:02:05.680004 12818 raft_consensus.cc:1275] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:02:05.671037 12863 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:05.683423 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.685128 12742 raft_consensus.cc:1240] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:05.658347 12748 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:05.720610 12824 raft_consensus.cc:2393] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:05.720542 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.653964 16205 raft_consensus.cc:2749] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:02:05.660744 16204 raft_consensus.cc:2749] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:05.694928 12750 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260504 20:02:05.675834 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.704500 16006 raft_consensus.cc:697] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:05.673038 16222 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:05.634413 16153 consensus_queue.cc:237] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.603189 16198 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:05.721560 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:05.771850 12748 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:05.590495 16223 consensus_queue.cc:1048] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20260504 20:02:05.693238 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.788597 12820 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.725574 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.730629 12893 raft_consensus.cc:1275] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:02:05.789359 12791 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.730538 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.790613 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.791378 12742 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260504 20:02:05.792606 12742 raft_consensus.cc:1240] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:05.754343 12899 raft_consensus.cc:1275] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260504 20:02:05.754597 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:02:05.754384 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 83472.7us
I20260504 20:02:05.759001 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:05.794299 12895 raft_consensus.cc:2393] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 9: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:02:05.760001 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:05.794791 12896 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:02:05.699916 16189 raft_consensus.cc:493] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:05.763914 13530 raft_consensus.cc:1240] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:05.759052 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.766114 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:05.766147 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.766095 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.719983 15605 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:05.799278 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.802829 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
W20260504 20:02:05.761626 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 64482.4us
I20260504 20:02:05.807698 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:05.808610 12742 raft_consensus.cc:1275] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260504 20:02:05.816316 12898 raft_consensus.cc:2468] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:02:05.810617 12891 raft_consensus.cc:2410] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260504 20:02:05.819118 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:05.819842 12893 raft_consensus.cc:2468] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 9.
I20260504 20:02:05.820748 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:05.824396 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.729175 16070 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:05.755784 16260 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:05.759156 16200 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 10 FOLLOWER]: Leader election won for term 10
I20260504 20:02:05.865151 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:05.865892 12714 leader_election.cc:400] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8.
I20260504 20:02:05.866283 12714 leader_election.cc:403] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:02:05.821425 12899 raft_consensus.cc:2468] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
W20260504 20:02:05.828037 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 174566us
I20260504 20:02:05.829392 12893 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:05.867871 12893 raft_consensus.cc:2468] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:05.769475 16222 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:05.765856 16169 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:05.886068 12820 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:02:05.889170 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:02:05.902107 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 318707us
I20260504 20:02:05.902474 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.902498 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.903401 12820 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:05.884938 16169 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:05.828517 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:05.858605 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.862643 12750 raft_consensus.cc:1275] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260504 20:02:05.912117 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 122850us
I20260504 20:02:05.863932 12741 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8. Ops: []
I20260504 20:02:05.724547 15834 consensus_queue.cc:1059] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20260504 20:02:05.870800 12714 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:05.723008 16217 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20260504 20:02:05.800228 16182 raft_consensus.cc:2749] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:05.873549 12716 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:05.874706 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:05.917332 12891 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:02:05.875634 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:05.918450 12898 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.821902 16189 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:05.907413 16006 consensus_queue.cc:237] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:05.878724 15651 raft_consensus.cc:2749] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 7. Current term is 8.
I20260504 20:02:05.910991 13499 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:05.927544 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.921715 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.923347 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:02:05.932739 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 62037.5us
I20260504 20:02:05.934640 16127 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:02:05.900580 16200 raft_consensus.cc:697] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 10 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:05.942103 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.942929 12825 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:05.945434 12895 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:02:05.852512 12748 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 4.
I20260504 20:02:05.877847 16178 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260504 20:02:05.908458 16169 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:05.872097 16260 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:05.961376 12747 raft_consensus.cc:1275] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260504 20:02:05.941253 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:05.970525 12714 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:05.938925 12820 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:02:05.980181 12820 raft_consensus.cc:1275] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:02:05.984001 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:05.985641 12820 raft_consensus.cc:1240] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:05.988112 12824 raft_consensus.cc:1240] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Rejecting Update request from peer ef6057d9a49e4942b47ea0fcd66ab6f7 for earlier term 10. Current term is 11. Ops: []
I20260504 20:02:05.914785 16185 consensus_queue.cc:1059] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260504 20:02:05.981616 15834 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:06.002669 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:06.004245 12820 raft_consensus.cc:1275] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:02:05.961436 16237 raft_consensus.cc:2804] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:06.018630 12818 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Advancing to term 10
W20260504 20:02:06.020092 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 146632us
I20260504 20:02:06.023030 12716 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:06.025038 12716 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:05.986984 15651 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader election won for term 6
W20260504 20:02:06.079704 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 54785.4us
I20260504 20:02:06.090497 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:05.982228 16250 consensus_queue.cc:1059] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.097s
I20260504 20:02:06.027294 16114 raft_consensus.cc:2749] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260504 20:02:06.112668 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 199663us
I20260504 20:02:06.084270 15951 raft_consensus.cc:2749] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:06.078634 16237 raft_consensus.cc:697] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:02:06.123611 12818 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 10.
I20260504 20:02:06.127234 12825 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:02:06.019902 15834 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:06.027397 16179 consensus_queue.cc:1048] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
I20260504 20:02:06.106540 16273 raft_consensus.cc:2749] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260504 20:02:06.159694 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 217320us
I20260504 20:02:06.161552 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:06.096127 16185 consensus_queue.cc:1059] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260504 20:02:06.117910 16198 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:06.167783 12819 raft_consensus.cc:1240] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
W20260504 20:02:06.168298 12715 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:06.132324 16242 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:06.162457 13530 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:06.169472 12898 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:02:06.188179 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:02:06.197829 16264 log.cc:927] Time spent T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
I20260504 20:02:06.183360 16272 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260504 20:02:06.204478 16275 consensus_queue.cc:1059] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
W20260504 20:02:06.206596 16263 log.cc:927] Time spent T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260504 20:02:06.207842 15834 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:06.186842 16242 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.209587 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:06.213694 16278 raft_consensus.cc:3055] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:06.214622 16260 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:06.216179 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:02:06.217655 16262 log.cc:927] Time spent T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.329s user 0.000s sys 0.001s
I20260504 20:02:06.113191 16200 consensus_queue.cc:237] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.226218 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:06.245653 13530 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
I20260504 20:02:06.311347 16242 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:06.300473 16265 log.cc:927] Time spent T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20260504 20:02:06.297910 16269 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
W20260504 20:02:06.323559 16215 consensus_meta.cc:287] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.700s user 0.008s sys 0.007s
W20260504 20:02:06.296989 16266 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.178s user 0.000s sys 0.001s
I20260504 20:02:06.334295 12715 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
W20260504 20:02:06.325989 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 209536us
I20260504 20:02:06.344754 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:06.345084 13526 raft_consensus.cc:2393] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 10: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:02:06.194854 16271 consensus_queue.cc:1059] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.152s
I20260504 20:02:06.229399 16171 consensus_queue.cc:1059] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
I20260504 20:02:06.354508 12750 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:02:06.370374 12864 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:06.249281 15651 raft_consensus.cc:697] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:06.372627 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:02:06.331362 16111 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.703s user 0.012s sys 0.000s
I20260504 20:02:06.250141 16278 raft_consensus.cc:740] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:06.369508 15951 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:02:06.353425 16215 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.432734 16111 raft_consensus.cc:2749] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:06.375857 16272 consensus_queue.cc:1059] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260504 20:02:06.257109 16237 consensus_queue.cc:237] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:06.448567 15951 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:06.458315 16070 raft_consensus.cc:2749] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 20:02:06.484244 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 403690us
I20260504 20:02:06.454993 16204 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260504 20:02:06.478780 12897 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8. Ops: []
W20260504 20:02:06.486636 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 472008us
I20260504 20:02:06.477584 15951 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:06.485664 12716 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:06.521842 12716 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:06.522467 12821 raft_consensus.cc:1275] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260504 20:02:06.524132 16272 consensus_queue.cc:1048] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:02:06.524982 12716 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:06.527025 16271 raft_consensus.cc:3055] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:02:06.527796 16271 raft_consensus.cc:740] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:06.529454 16271 consensus_queue.cc:260] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:06.541766 16268 log.cc:927] Time spent T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.312s user 0.001s sys 0.000s
W20260504 20:02:06.543030 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 170521us
I20260504 20:02:06.543846 16162 raft_consensus.cc:3055] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Stepping down as leader of term 9
I20260504 20:02:06.544380 16162 raft_consensus.cc:740] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:06.547545 12716 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:06.553957 12716 leader_election.cc:304] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
W20260504 20:02:06.542601 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 108323us
I20260504 20:02:06.510171 16070 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:02:06.476626 16273 consensus_queue.cc:1048] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260504 20:02:06.545727 16114 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:06.563637 12791 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:06.542716 16205 raft_consensus.cc:3055] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:02:06.545068 15993 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:06.577677 12716 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:06.579349 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:06.580345 12742 raft_consensus.cc:2410] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260504 20:02:06.571296 13497 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:06.572515 16285 raft_consensus.cc:2749] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:06.573251 12743 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260504 20:02:06.574329 12748 raft_consensus.cc:1275] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:02:06.566542 16070 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:06.587795 16070 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:06.588894 16188 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:06.589464 16188 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.560870 16127 raft_consensus.cc:2749] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:06.554404 16171 raft_consensus.cc:2749] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:02:06.610378 12750 raft_consensus.cc:1275] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:02:06.585049 12748 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260504 20:02:06.586720 16299 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:06.587046 16149 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260504 20:02:06.612685 16299 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.615338 16299 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:06.587209 16140 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:06.616374 16140 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:06.618768 16140 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:06.592393 13497 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:02:06.575474 15605 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.749s user 0.010s sys 0.005s
I20260504 20:02:06.612174 16171 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:06.622336 16171 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:06.622793 16171 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:06.625905 16203 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:06.626577 16203 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:06.590901 16003 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.878s user 0.013s sys 0.000s
I20260504 20:02:06.628929 12749 raft_consensus.cc:2410] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260504 20:02:06.632784 16149 raft_consensus.cc:2749] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:02:06.621739 13497 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:06.636389 13497 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:06.637356 16149 raft_consensus.cc:2749] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260504 20:02:06.554716 16111 consensus_queue.cc:1048] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260504 20:02:06.647500 12899 raft_consensus.cc:1275] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260504 20:02:06.650319 12891 raft_consensus.cc:1275] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260504 20:02:06.483270 16278 consensus_queue.cc:260] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:06.654096 13497 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:06.577750 15993 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:06.650024 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:06.657929 12748 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 9.
I20260504 20:02:06.659432 12865 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:06.578004 16205 raft_consensus.cc:740] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:06.517906 15651 consensus_queue.cc:237] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:06.610395 16169 consensus_meta.cc:287] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.617s user 0.001s sys 0.012s
I20260504 20:02:06.679102 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:06.671661 16215 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:06.671440 16305 consensus_queue.cc:1059] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
W20260504 20:02:06.678814 12864 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 165289us
I20260504 20:02:06.677557 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:06.687279 12748 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:06.667884 15993 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:06.649570 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 71990.4us
I20260504 20:02:06.690141 12865 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:06.677278 16149 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260504 20:02:06.702701 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:06.703444 12899 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:02:06.706837 12899 raft_consensus.cc:1275] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260504 20:02:06.707772 12895 raft_consensus.cc:1275] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260504 20:02:06.707525 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:06.686595 16140 consensus_queue.cc:1059] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20260504 20:02:06.721282 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:06.709501 16299 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:06.712723 16310 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:06.713374 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:06.716686 16171 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.681613 15943 raft_consensus.cc:2749] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260504 20:02:06.722379 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:06.722687 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:06.610179 16162 consensus_queue.cc:260] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.698046 16300 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:02:06.683854 16169 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:06.737972 12895 raft_consensus.cc:3055] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:02:06.749138 12895 raft_consensus.cc:740] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:06.698529 16215 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:06.741761 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:06.732829 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:06.721458 16188 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:06.762599 12895 consensus_queue.cc:260] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.764055 12895 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:06.767659 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:06.769429 12898 raft_consensus.cc:1275] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260504 20:02:06.771322 12898 raft_consensus.cc:1275] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260504 20:02:06.733718 16299 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:06.781352 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 50146.9us
W20260504 20:02:06.784683 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:06.791352 12896 raft_consensus.cc:2368] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7.
I20260504 20:02:06.746430 16300 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:06.803704 12716 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:06.805263 13499 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:06.807011 12895 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
W20260504 20:02:06.811190 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 71457.4us
I20260504 20:02:06.779312 16162 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:06.780479 16299 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:06.817317 16186 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260504 20:02:06.765278 16215 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.746440 16205 consensus_queue.cc:260] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.813176 16303 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:06.812305 16300 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:06.818113 16204 raft_consensus.cc:2749] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:06.860325 12897 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1879) took 1132 ms. Trace:
I20260504 20:02:06.739761 16310 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.812884 15834 consensus_queue.cc:1048] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260504 20:02:06.867748 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1886) took 1293 ms. Trace:
I20260504 20:02:06.826009 16306 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:06.831355 16291 consensus_queue.cc:1048] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.206s
I20260504 20:02:06.844463 16318 consensus_queue.cc:1048] T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.131s
I20260504 20:02:06.911551 13528 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260504 20:02:06.834379 16319 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:06.837085 16215 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:06.856088 16292 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:06.955897 12713 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:06.961226 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:06.828811 16302 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election won for term 4
I20260504 20:02:06.962852 12896 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1870) took 1238 ms. Trace:
I20260504 20:02:06.964663 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:02:06.970839 12894 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1863) took 1248 ms (client timeout 1526 ms). Trace:
I20260504 20:02:06.963758 12749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1932) took 1283 ms. Trace:
I20260504 20:02:06.972172 12749 rpcz_store.cc:276] 0504 20:02:05.680563 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.682836 (+ 2273us) service_pool.cc:224] Handling call
0504 20:02:06.963498 (+1280662us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b5400028d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:06.963693 (+ 195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1919707392}
I20260504 20:02:06.969678 12896 rpcz_store.cc:276] 0504 20:02:05.724851 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.822117 (+ 97266us) service_pool.cc:224] Handling call
0504 20:02:06.962632 (+1140515us) spinlock_profiling.cc:243] Waited 925 ms on lock 0x7b540013bfd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:06.962784 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2128198016}
W20260504 20:02:06.971321 12894 rpcz_store.cc:269] 0504 20:02:05.722561 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.723749 (+ 1188us) service_pool.cc:224] Handling call
0504 20:02:06.964480 (+1240731us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:02:06.970754 (+ 6274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8729472}
W20260504 20:02:06.983670 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 171467us
I20260504 20:02:06.867913 16303 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:06.964752 16319 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.987574 16169 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:06.964409 16289 log.cc:927] Time spent T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.294s user 0.002s sys 0.000s
W20260504 20:02:06.988855 16290 log.cc:927] Time spent T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.500s user 0.003s sys 0.000s
W20260504 20:02:06.990347 16293 log.cc:927] Time spent T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.224s user 0.002s sys 0.000s
I20260504 20:02:06.993300 12819 raft_consensus.cc:3055] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 LEADER]: Stepping down as leader of term 8
W20260504 20:02:06.950204 16288 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.272s user 0.000s sys 0.002s
I20260504 20:02:06.993835 12819 raft_consensus.cc:740] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:06.995288 16205 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:06.995640 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:06.996426 12892 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 5.
I20260504 20:02:06.997035 12892 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37318 (request call id 1866) took 1273 ms. Trace:
I20260504 20:02:06.997890 16185 consensus_queue.cc:1048] T 7ddedcb68670423f9dbf822898141e0f P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260504 20:02:06.997416 12892 rpcz_store.cc:276] 0504 20:02:05.723635 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.725974 (+ 2339us) service_pool.cc:224] Handling call
0504 20:02:06.995431 (+1269457us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161982760 00007f0161982044 00007f015aeef834 00007f015aeef7c2
0504 20:02:06.996956 (+ 1525us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8942464}
I20260504 20:02:06.952154 16267 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260504 20:02:06.954495 16308 log.cc:927] Time spent T 4318d305513d467c8254d7c054ef993e P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20260504 20:02:06.994809 12819 consensus_queue.cc:260] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:06.861532 16147 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:06.989456 16319 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:06.900449 16309 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:06.985031 16280 raft_consensus.cc:3055] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 LEADER]: Stepping down as leader of term 10
I20260504 20:02:07.014691 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:07.274969 12890 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38968 (request call id 614) took 1077 ms. Trace:
I20260504 20:02:07.595496 12890 rpcz_store.cc:276] 0504 20:02:06.197391 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.197620 (+ 229us) service_pool.cc:224] Handling call
0504 20:02:07.274134 (+1076514us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b58001d3498. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:02:07.274384 (+ 250us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:07.274635 (+ 251us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:07.274659 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:07.274701 (+ 42us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:02:07.274733 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:07.274792 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:07.274899 (+ 107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47727872}
I20260504 20:02:07.258720 12741 raft_consensus.cc:1240] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 LEADER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:07.138607 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:02:07.085287 12824 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.001s sys 0.021s
I20260504 20:02:07.015411 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 2224) took 1306 ms. Trace:
W20260504 20:02:07.597836 12824 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.031s user 0.008s sys 0.037s
I20260504 20:02:06.881423 12744 rpcz_store.cc:276] 0504 20:02:05.574597 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.577911 (+ 3314us) service_pool.cc:224] Handling call
0504 20:02:05.612338 (+ 34427us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:05.636864 (+ 24526us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:05.636895 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:05.665955 (+ 29060us) log.cc:844] Serialized 26 byte log entry
0504 20:02:05.898188 (+232233us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:02:05.898229 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:06.864263 (+966034us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b5400027454. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:06.864315 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:06.864371 (+ 56us) raft_consensus.cc:1685] finished
0504 20:02:06.864385 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:06.867661 (+ 3276us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":642,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":13172,"spinlock_wait_cycles":55500544,"thread_start_us":191977,"threads_started":1,"wal-append.queue_time_us":309242,"wal-append.run_cpu_time_us":6274,"wal-append.run_wall_time_us":1459980}
W20260504 20:02:07.398145 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 441688us
I20260504 20:02:06.833392 16114 raft_consensus.cc:2749] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:02:06.876667 12897 rpcz_store.cc:276] 0504 20:02:05.728325 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.900889 (+172564us) service_pool.cc:224] Handling call
0504 20:02:05.919861 (+ 18972us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:06.486813 (+566952us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:06.859967 (+373154us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b540013c9d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:06.860230 (+ 263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1284771712}
W20260504 20:02:07.160619 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 148119us
W20260504 20:02:07.561975 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 758341us
W20260504 20:02:07.568454 15993 consensus_meta.cc:287] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.866s user 0.013s sys 0.003s
I20260504 20:02:07.448859 11536 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260504 20:02:07.514467 12818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 379) took 1785 ms (client timeout 1607 ms). Trace:
I20260504 20:02:07.332116 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1889) took 1426 ms. Trace:
W20260504 20:02:07.331233 16287 log.cc:927] Time spent T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.755s user 0.000s sys 0.002s
W20260504 20:02:07.336663 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 991592us
I20260504 20:02:07.607438 12747 rpcz_store.cc:276] 0504 20:02:05.905970 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.906186 (+ 216us) service_pool.cc:224] Handling call
0504 20:02:05.907922 (+ 1736us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:05.975579 (+ 67657us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:07.331729 (+1356150us) spinlock_profiling.cc:243] Waited 46.6 ms on lock 0x7b54000dc5d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:07.332023 (+ 294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":107209600}
W20260504 20:02:07.377414 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:07.538103 16301 raft_consensus.cc:3055] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Stepping down as leader of term 7
W20260504 20:02:07.289758 16222 consensus_meta.cc:287] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.392s user 0.013s sys 0.005s
W20260504 20:02:07.584156 16237 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.004s sys 0.004s
W20260504 20:02:07.317798 16189 consensus_meta.cc:287] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.375s user 0.013s sys 0.005s
I20260504 20:02:07.619007 12749 raft_consensus.cc:1275] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:02:06.954805 16311 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.598709 13528 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:06.977973 16215 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:02:07.602699 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:07.011706 16267 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:07.165874 16302 raft_consensus.cc:697] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:07.162181 16216 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
W20260504 20:02:07.247264 16200 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.014s sys 0.015s
I20260504 20:02:07.609512 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:02:07.607177 12818 rpcz_store.cc:269] 0504 20:02:05.710744 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.718909 (+ 8165us) service_pool.cc:224] Handling call
0504 20:02:07.495789 (+1776880us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54000e4cd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:07.495985 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":477603712}
I20260504 20:02:06.993608 16310 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:07.597980 12817 rpcz_store.cc:276] 0504 20:02:05.700260 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.700921 (+ 661us) service_pool.cc:224] Handling call
0504 20:02:05.702871 (+ 1950us) raft_consensus.cc:1471] Updating replica for 2 ops
0504 20:02:06.129315 (+426444us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:06.129349 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0504 20:02:06.153754 (+ 24405us) log.cc:844] Serialized 52 byte log entry
0504 20:02:06.436142 (+282388us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:02:06.436201 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:06.436310 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:07.006705 (+570395us) raft_consensus.cc:1685] finished
0504 20:02:07.006731 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:07.006920 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41213,"raft.queue_time_us":203066,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":10605,"thread_start_us":613839,"threads_started":2,"wal-append.queue_time_us":311645}
I20260504 20:02:06.988857 16171 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:07.616490 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:07.620731 12742 raft_consensus.cc:1275] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:02:07.141254 15834 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:07.271646 16285 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.247545 16204 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.141952 16070 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.483799 16182 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:07.084482 16158 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:07.061575 16327 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:02:07.625859 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 616973us
W20260504 20:02:07.618657 12696 auto_rebalancer.cc:806] Removing replace marker failed: Replica 5ff0ef72ec3b4667893a0420d3c7f923 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "82787d8e64d24ca885ca517aa8ed1be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } }
I20260504 20:02:07.629546 16272 consensus_queue.cc:1048] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260504 20:02:07.629370 16215 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.595067 16280 raft_consensus.cc:740] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:02:07.665609 12817 raft_consensus.cc:1240] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:07.666468 12746 raft_consensus.cc:1240] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:07.066497 16303 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:06.981775 16306 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:06.982093 16292 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:07.616416 16237 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s user 0.005s sys 0.005s
I20260504 20:02:07.006498 15951 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.633761 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:07.635356 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:07.611600 16319 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.774327 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:07.611229 16162 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:07.616694 16300 consensus_meta.cc:287] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.736s user 0.013s sys 0.002s
I20260504 20:02:07.666221 16186 raft_consensus.cc:493] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.613332 16301 raft_consensus.cc:740] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:02:07.646512 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:07.604518 12716 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:02:07.793071 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 188640us
W20260504 20:02:07.632347 16200 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.958s user 0.030s sys 0.032s
I20260504 20:02:07.621286 16222 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.652664 13528 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:07.630963 16267 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:07.624017 16236 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:07.664494 16302 consensus_queue.cc:237] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.605450 16147 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.643376 16203 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.602119 16309 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:07.666734 12696 auto_rebalancer.cc:814] Could not move replica: Incomplete: tablet c9a8aee69ec8499895f3ab719968d204, TS 82787d8e64d24ca885ca517aa8ed1be2 -> TS ef6057d9a49e4942b47ea0fcd66ab6f7 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:07.656517 16328 log.cc:927] Time spent T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260504 20:02:07.670483 12891 raft_consensus.cc:2368] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5.
I20260504 20:02:07.649268 16253 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.652062 15834 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.873008 16309 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:07.873466 15834 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:07.836484 16280 consensus_queue.cc:260] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.874090 15834 raft_consensus.cc:493] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:07.624761 16189 raft_consensus.cc:515] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.874665 16280 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:07.757648 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:07.875540 12822 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:07.733563 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:07.876416 12817 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:07.734652 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:07.877477 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:07.775220 12821 raft_consensus.cc:2468] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:07.878225 12817 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:07.878813 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:02:07.775322 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 135879us
I20260504 20:02:07.879571 16326 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.775826 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:07.880115 16326 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.880829 13526 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 10.
I20260504 20:02:07.779081 12745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1929) took 2099 ms. Trace:
W20260504 20:02:07.758778 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 99836.7us
I20260504 20:02:07.882460 12791 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:07.881680 12745 rpcz_store.cc:276] 0504 20:02:05.679417 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.681976 (+ 2559us) service_pool.cc:224] Handling call
0504 20:02:05.694135 (+ 12159us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:05.768909 (+ 74774us) raft_consensus.cc:1534] Early marking committed up to index 5
0504 20:02:06.129691 (+360782us) log.cc:844] Serialized 14 byte log entry
0504 20:02:06.436546 (+306855us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:06.453779 (+ 17233us) log.cc:844] Serialized 26 byte log entry
0504 20:02:06.454016 (+ 237us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:02:06.454082 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:06.454201 (+ 119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:07.330155 (+875954us) spinlock_profiling.cc:243] Waited 6.98 ms on lock 0x7b440002534c. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f015981d375 00007f015981dd92 00007f015c55957b 00007f015c554e4f 00007f015c548c0e 00007f015c547f56 00007f015c547b78 00007f015c5515c1 00007f015c551579 00007f015c551509
0504 20:02:07.754786 (+424631us) raft_consensus.cc:1685] finished
0504 20:02:07.778787 (+ 24001us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:07.779002 (+ 215us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":495395,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":48688,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":42589,"spinlock_wait_cycles":16045312,"thread_start_us":530866,"threads_started":2,"wal-append.queue_time_us":417502}
W20260504 20:02:07.866487 12696 auto_rebalancer.cc:295] scheduled replica move failed to complete: Incomplete: tablet c9a8aee69ec8499895f3ab719968d204, TS 82787d8e64d24ca885ca517aa8ed1be2 -> TS ef6057d9a49e4942b47ea0fcd66ab6f7 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.614483 15993 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.883968 12791 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:07.885281 15993 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:07.787753 12816 raft_consensus.cc:3060] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:07.753134 16333 log.cc:927] Time spent T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260504 20:02:07.894793 16306 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:07.754469 16038 raft_consensus.cc:493] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.677563 16034 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.099s
I20260504 20:02:07.731879 16335 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.896378 16038 raft_consensus.cc:515] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.896570 16335 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.826805 12716 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:07.656840 16204 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.828632 13530 raft_consensus.cc:1240] T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:07.752482 16292 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:07.642992 16216 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.898890 16038 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:07.899902 16336 log.cc:927] Time spent T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20260504 20:02:07.900939 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 74255us
I20260504 20:02:07.901182 16216 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:07.785887 16178 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.902221 16147 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.902045 16178 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.794862 16186 raft_consensus.cc:515] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.904341 16292 raft_consensus.cc:2749] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:07.904848 16178 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.660892 16320 consensus_queue.cc:1048] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.136s
I20260504 20:02:07.780548 16171 raft_consensus.cc:493] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:07.630753 16311 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.909407 16171 raft_consensus.cc:515] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.898820 16335 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:07.911563 16311 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.656301 16329 consensus_queue.cc:1048] T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.146s
I20260504 20:02:07.913147 16235 raft_consensus.cc:2764] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260504 20:02:07.660804 16182 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.825253 16086 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.914129 16250 raft_consensus.cc:2804] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:02:07.914722 16250 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:07.914530 16086 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.915264 16250 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:02:07.893685 16299 consensus_meta.cc:287] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 1.064s user 0.012s sys 0.003s
I20260504 20:02:07.916287 16299 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.666538 16158 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.837713 12893 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [term 10 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260504 20:02:07.839226 12741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1893) took 1618 ms. Trace:
I20260504 20:02:07.924274 12741 rpcz_store.cc:276] 0504 20:02:06.220439 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.220715 (+ 276us) service_pool.cc:224] Handling call
0504 20:02:06.245914 (+ 25199us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:07.597104 (+1351190us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:07.838953 (+241849us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400027e54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:07.839159 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2328610560}
I20260504 20:02:07.839401 13527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:07.846334 15951 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:07.847668 16215 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:07.933117 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 52716.6us
I20260504 20:02:07.788574 16162 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.851296 13529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:07.853255 13533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:07.853636 16070 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.656908 16285 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.787346 16319 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.827817 16261 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.788607 16300 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.786886 16303 consensus_queue.cc:1048] T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.048s
I20260504 20:02:07.854764 16301 consensus_queue.cc:260] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.667066 16327 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.855376 16236 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.972958 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce60695d3db4c06ad8e0eec0419f79f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:08.002609 12897 raft_consensus.cc:3060] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:07.825551 16153 raft_consensus.cc:493] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:07.867965 16114 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.868075 16318 raft_consensus.cc:493] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:07.867635 16339 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:07.871618 16337 log.cc:927] Time spent T 1ebe61f64fd74c158829e717ae37a5b9 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20260504 20:02:07.871224 16253 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:07.874763 12891 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2118) took 1166 ms. Trace:
I20260504 20:02:07.886646 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2229) took 1397 ms. Trace:
I20260504 20:02:07.888760 12715 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:07.890056 12791 leader_election.cc:304] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:07.874619 15834 raft_consensus.cc:515] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:07.891613 16222 leader_election.cc:290] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.893126 16189 leader_election.cc:290] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.779330 16273 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.905033 16186 leader_election.cc:290] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:07.853308 16205 consensus_meta.cc:287] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.857s user 0.009s sys 0.004s
I20260504 20:02:07.906433 15993 consensus_queue.cc:1048] T 85ec7d979c6d41db891f52c7e0bbcefe P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:02:07.922082 16158 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.919930 16086 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:07.933015 13527 raft_consensus.cc:2393] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in this term.
I20260504 20:02:07.795277 12743 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260504 20:02:07.929415 12863 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:07.939690 12822 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:07.941903 12816 raft_consensus.cc:2410] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:02:07.942209 13533 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Advancing to term 10
W20260504 20:02:07.946009 16241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.008s sys 0.009s
I20260504 20:02:07.979103 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
W20260504 20:02:07.966686 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 306328us
I20260504 20:02:07.922569 16292 consensus_queue.cc:1048] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.022s
W20260504 20:02:08.323760 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 321388us
I20260504 20:02:08.293434 12897 raft_consensus.cc:2468] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:02:07.829984 16310 raft_consensus.cc:2749] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:07.969029 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:07.981611 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:07.991119 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:07.944388 15951 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.946722 16215 raft_consensus.cc:493] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:08.318912 12824 rpcz_store.cc:276] 0504 20:02:06.489131 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.489264 (+ 133us) service_pool.cc:224] Handling call
0504 20:02:06.489638 (+ 374us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:06.490107 (+ 469us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:06.490139 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:06.497899 (+ 7760us) log.cc:844] Serialized 26 byte log entry
0504 20:02:07.598330 (+1100431us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:02:07.598378 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:07.598445 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:07.886418 (+287973us) raft_consensus.cc:1685] finished
0504 20:02:07.886443 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:07.886581 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":73147,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":2715,"thread_start_us":1031886,"threads_started":1,"wal-append.queue_time_us":1017633}
W20260504 20:02:08.244261 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:08.436890 12746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1936) took 2686 ms. Trace:
W20260504 20:02:08.438678 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:08.319501 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 390154us
I20260504 20:02:08.320549 12891 rpcz_store.cc:276] 0504 20:02:06.708344 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.735198 (+ 26854us) service_pool.cc:224] Handling call
0504 20:02:07.874544 (+1139346us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b540013bad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:07.874696 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2114773888}
I20260504 20:02:08.438596 12746 rpcz_store.cc:276] 0504 20:02:05.698578 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.791840 (+ 93262us) service_pool.cc:224] Handling call
0504 20:02:05.792224 (+ 384us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:07.667038 (+1874814us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:08.384576 (+717538us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54002bd4d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:08.384881 (+ 305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4309947136}
I20260504 20:02:08.035677 12896 raft_consensus.cc:1275] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260504 20:02:08.334568 13499 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:08.114418 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:07.944175 16187 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:08.063397 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:08.321506 12894 raft_consensus.cc:2468] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:02:07.606753 11536 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260504 20:02:08.441993 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:08.351727 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:02:08.243043 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 151304us
I20260504 20:02:08.440129 12863 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:08.441053 12892 raft_consensus.cc:2468] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
I20260504 20:02:07.952643 16038 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:08.422478 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:08.388549 12742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1950) took 1804 ms. Trace:
I20260504 20:02:07.985131 16329 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:08.319172 16241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.908s user 0.009s sys 0.009s
I20260504 20:02:08.449537 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:08.447472 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:08.450769 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:08.448478 12742 rpcz_store.cc:276] 0504 20:02:06.584103 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.584999 (+ 896us) service_pool.cc:224] Handling call
0504 20:02:06.585274 (+ 275us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:07.642979 (+1057705us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:08.388152 (+745173us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000dcad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:08.388443 (+ 291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2380542848}
I20260504 20:02:08.445653 12899 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:02:08.349370 15943 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:07.986829 16070 consensus_queue.cc:1048] T faf44d4cfb914599a82b50e6d195ecbd P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.476s
I20260504 20:02:08.462417 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:08.320443 16299 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:08.463406 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:08.242547 16311 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:08.462417 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:08.240125 16216 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:08.468712 12894 raft_consensus.cc:2393] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:02:08.470404 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:08.470404 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:08.489108 12824 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 3.
I20260504 20:02:08.049324 16301 raft_consensus.cc:3060] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:08.489281 12822 raft_consensus.cc:2410] T 85ec7d979c6d41db891f52c7e0bbcefe P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260504 20:02:08.092466 16306 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:08.001240 16309 consensus_queue.cc:1048] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.471s
I20260504 20:02:08.349072 16335 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:08.350855 15605 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:02:08.427668 16326 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.446509 16038 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:08.489166 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 599184us
I20260504 20:02:08.490747 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:08.504707 12824 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 9.
I20260504 20:02:08.492594 12864 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:08.505300 12791 leader_election.cc:304] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:08.466967 16303 raft_consensus.cc:2804] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:02:08.472046 16278 raft_consensus.cc:3060] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:08.501684 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:08.008090 16235 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:08.451566 15951 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.379509 16114 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.509198 12791 leader_election.cc:304] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:08.383666 16153 raft_consensus.cc:515] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.478305 15943 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.497751 15605 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
W20260504 20:02:08.513294 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 73273.1us
I20260504 20:02:08.451099 16215 raft_consensus.cc:515] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.001931 16261 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.423961 16339 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.492434 16311 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:08.511157 16267 consensus_meta.cc:287] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.673s user 0.015s sys 0.000s
I20260504 20:02:08.502970 16158 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:08.349303 16273 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.510255 16303 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:08.444132 16318 raft_consensus.cc:515] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.513442 16300 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:08.586575 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 44905.3us
I20260504 20:02:08.453795 16187 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.474263 16329 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.552965 16253 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.561048 16182 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:08.498811 16306 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.524928 16189 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:08.526006 16310 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260504 20:02:08.537720 16086 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:08.499173 16216 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.527493 16275 raft_consensus.cc:2804] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:08.679458 12895 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:02:08.649075 16285 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:08.538064 15605 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:08.507017 16335 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.592046 16215 leader_election.cc:290] T 6673c1682edf457ba633ba714a47587f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:08.594282 16038 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:08.551529 16350 raft_consensus.cc:2804] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:08.581326 16267 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.604750 15834 leader_election.cc:290] T e44b757a449142a38ed04025e01cb4a5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:08.605253 16280 consensus_meta.cc:287] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.730s user 0.011s sys 0.001s
I20260504 20:02:08.536793 16162 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:08.646476 12714 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.009s sys 0.074s
W20260504 20:02:08.699561 12714 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.010s sys 0.074s
I20260504 20:02:08.672650 16327 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:08.652491 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 205360us
I20260504 20:02:08.526443 16235 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.572336 16236 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:08.583364 16303 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:08.687573 16086 raft_consensus.cc:697] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:08.583416 16319 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:08.650070 16300 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:08.686730 12890 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.001s sys 0.005s
W20260504 20:02:08.718214 12890 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.001s sys 0.005s
I20260504 20:02:08.582389 16158 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.659968 16186 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:08.694814 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 298538us
I20260504 20:02:08.668099 16189 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:08.579916 13533 raft_consensus.cc:2468] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 10.
I20260504 20:02:08.671239 16310 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:08.700007 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 694974us
W20260504 20:02:08.650967 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 62700.8us
I20260504 20:02:08.728667 12791 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:08.705263 13532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:08.724597 12863 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:08.732009 13532 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:02:08.684062 16275 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:08.707023 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:02:08.707688 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:08.716616 13524 raft_consensus.cc:1240] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5. Ops: []
I20260504 20:02:08.708091 13529 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 11.
I20260504 20:02:08.694468 16350 raft_consensus.cc:493] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:08.722332 13526 raft_consensus.cc:1240] T 410644ff9c0449188275a6d49ecc0c2d P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:08.724072 12716 leader_election.cc:304] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:08.725214 13499 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:08.727246 13528 raft_consensus.cc:1240] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: [9.10-9.10]
W20260504 20:02:08.744551 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:08.763743 12863 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:08.758606 12716 leader_election.cc:400] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5.
I20260504 20:02:08.766881 12716 leader_election.cc:403] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:02:08.757064 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:08.720674 16327 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:08.769626 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:08.772015 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 46886.8us
I20260504 20:02:08.774068 16273 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.773947 16253 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260504 20:02:08.767113 16338 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:08.776515 16338 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.727684 16186 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.778147 16158 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.779193 16357 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:08.780604 16186 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:08.768721 15834 raft_consensus.cc:2804] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:08.781956 15834 raft_consensus.cc:697] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:02:08.775753 16348 log.cc:927] Time spent T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260504 20:02:08.783337 16351 log.cc:927] Time spent T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260504 20:02:08.783531 15834 consensus_queue.cc:237] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.786120 16267 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.786248 16285 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:08.787107 16285 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:08.775213 16086 consensus_queue.cc:237] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:08.825932 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 54607.1us
I20260504 20:02:08.827191 12791 leader_election.cc:400] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7.
I20260504 20:02:08.827602 12791 leader_election.cc:403] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260504 20:02:08.730280 16189 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:02:08.837348 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 67351.7us
I20260504 20:02:08.758522 16309 raft_consensus.cc:493] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:08.738809 16310 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Advancing to term 11
W20260504 20:02:08.791193 16250 consensus_meta.cc:287] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.875s user 0.006s sys 0.011s
I20260504 20:02:08.726440 16300 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:08.768770 16345 log.cc:927] Time spent T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.163s user 0.001s sys 0.001s
I20260504 20:02:08.837500 16182 consensus_queue.cc:1048] T f84c093f7c744182b34146f8f1e644d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260504 20:02:08.861325 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:08.728233 15951 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.769467 16275 raft_consensus.cc:3060] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:08.944416 16306 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.845981 12714 leader_election.cc:304] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
W20260504 20:02:08.946578 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 100785us
I20260504 20:02:08.795928 15943 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.893236 12716 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:08.757611 16350 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:08.791657 16261 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.794376 16329 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:08.878130 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 105317us
W20260504 20:02:08.900743 12864 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:08.770777 16114 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.774092 16187 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:08.954528 12821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.007s sys 0.018s
W20260504 20:02:08.983546 12821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.893s user 0.008s sys 0.020s
I20260504 20:02:08.770751 16162 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election won for term 6
I20260504 20:02:08.803450 16236 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:02:08.932267 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:02:08.933175 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 99594.3us
I20260504 20:02:08.841742 12791 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:08.925442 12715 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.004s sys 0.079s
I20260504 20:02:08.790604 16319 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:02:08.939038 16300 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:08.941982 16311 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:08.788030 16335 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.789187 16216 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:08.791383 16347 log.cc:927] Time spent T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20260504 20:02:08.939085 16318 leader_election.cc:290] T 445dd93bcaf04bae88b84d4eb9eddb60 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:08.832978 16153 leader_election.cc:290] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:08.979138 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 86010.4us
W20260504 20:02:08.986979 12715 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s user 0.006s sys 0.079s
W20260504 20:02:09.016054 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 1.12736e+06us
I20260504 20:02:09.016214 12716 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:08.843604 16352 raft_consensus.cc:2749] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7.
W20260504 20:02:08.987180 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 145574us
I20260504 20:02:08.984584 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2244) took 1195 ms. Trace:
I20260504 20:02:09.019555 12791 leader_election.cc:304] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:08.993583 12863 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.68.2:41415 timed out after 1.892s (SENT) [suppressed 28 similar messages]
I20260504 20:02:09.019371 12821 rpcz_store.cc:276] 0504 20:02:07.789081 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.890968 (+101887us) service_pool.cc:224] Handling call
0504 20:02:07.891255 (+ 287us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:07.891457 (+ 202us) raft_consensus.cc:1534] Early marking committed up to index 4
0504 20:02:08.090450 (+198993us) log.cc:844] Serialized 14 byte log entry
0504 20:02:08.984245 (+893795us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:08.984317 (+ 72us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:02:08.984365 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:08.984420 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:08.984533 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":592863,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":1440,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":8184,"thread_start_us":1091912,"threads_started":2}
I20260504 20:02:08.791513 16341 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:08.829918 16327 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:08.995594 16236 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.008505 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:08.831629 16322 consensus_queue.cc:1048] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.154s
I20260504 20:02:09.013337 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:08.957190 15993 raft_consensus.cc:2749] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260504 20:02:09.024438 12896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:09.007043 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:08.975816 16235 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.999770 16261 raft_consensus.cc:2749] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:08.933933 16250 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.055713 16250 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:08.995028 16217 raft_consensus.cc:2804] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:09.056763 16217 raft_consensus.cc:493] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.015053 15951 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.028393 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.028287 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.030858 12742 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 6.
I20260504 20:02:09.058048 16182 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.058811 16182 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.058965 16318 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.059092 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.059895 12742 raft_consensus.cc:2468] T 7e652e6edc2c4497b0f7f50dc79598f2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 10.
I20260504 20:02:09.059754 16318 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.058995 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.061056 16182 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.061456 12750 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:09.026330 16038 raft_consensus.cc:493] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.062389 16038 raft_consensus.cc:515] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.064468 16038 leader_election.cc:290] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:09.067694 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6673c1682edf457ba633ba714a47587f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.070261 16362 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:02:09.071516 16361 log.cc:927] Time spent T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260504 20:02:09.071283 16362 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.073244 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2245) took 1281 ms. Trace:
I20260504 20:02:09.073378 12750 raft_consensus.cc:2468] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:09.073671 12824 rpcz_store.cc:276] 0504 20:02:07.791833 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:08.510772 (+718939us) service_pool.cc:224] Handling call
0504 20:02:08.534597 (+ 23825us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:08.548199 (+ 13602us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:08.548255 (+ 56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:08.552179 (+ 3924us) log.cc:844] Serialized 26 byte log entry
0504 20:02:08.717330 (+165151us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:02:08.717373 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:08.717450 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:09.072971 (+355521us) raft_consensus.cc:1685] finished
0504 20:02:09.072997 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.073164 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12997,"raft.queue_time_us":22772,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":169,"thread_start_us":113512,"threads_started":1,"wal-append.queue_time_us":368463}
I20260504 20:02:09.075233 16362 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.075904 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.038779 12897 raft_consensus.cc:1275] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:02:09.077520 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.079067 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.079941 16326 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.080389 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5409cb132bc14d1cafaa29b22ced9dc9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.038682 12892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.058199 12891 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:02:08.926362 16309 raft_consensus.cc:515] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.043105 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:08.957580 16339 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.048491 12896 raft_consensus.cc:3060] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:09.049913 12899 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:09.045612 16152 raft_consensus.cc:2804] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 20:02:09.057348 16217 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:09.029119 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:02:09.069628 16278 consensus_meta.cc:287] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.557s user 0.005s sys 0.006s
I20260504 20:02:09.083875 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.128790 12746 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:09.130123 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ea905fd808496c9b7d84a57c496cb3" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.130909 12746 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:09.042399 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.003523 16162 raft_consensus.cc:697] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Leader. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:09.037688 12894 raft_consensus.cc:1240] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:02:09.025713 16341 raft_consensus.cc:493] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:09.043964 16236 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:09.061287 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:08.997372 16319 raft_consensus.cc:493] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.047158 16353 raft_consensus.cc:2804] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260504 20:02:09.109597 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:09.149498 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:09.150213 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:09.029392 16285 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.119899 12821 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 10
I20260504 20:02:09.155416 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.076036 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.120312 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.131374 12745 raft_consensus.cc:1240] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260504 20:02:09.083891 12750 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:09.080793 16326 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.142331 16314 raft_consensus.cc:493] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.141927 16162 consensus_queue.cc:237] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.147544 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e44b757a449142a38ed04025e01cb4a5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
W20260504 20:02:09.101708 16347 log.cc:927] Time spent T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260504 20:02:09.151194 12746 raft_consensus.cc:2468] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 9.
W20260504 20:02:09.163880 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 166517us
I20260504 20:02:09.119333 16152 raft_consensus.cc:493] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:09.174682 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.168725 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.118944 16362 raft_consensus.cc:493] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:09.169219 12897 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.206311 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2241) took 1450 ms. Trace:
I20260504 20:02:09.170400 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.206658 16362 raft_consensus.cc:515] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.206830 12897 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:02:09.206825 12817 rpcz_store.cc:276] 0504 20:02:07.755311 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.878898 (+123587us) service_pool.cc:224] Handling call
0504 20:02:07.879206 (+ 308us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:08.562905 (+683699us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:08.562954 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:08.563017 (+ 63us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:02:08.563076 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.205530 (+642454us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b540015e0d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.205588 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.205761 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1571864960}
I20260504 20:02:09.208974 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:09.184063 16152 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:09.207387 12824 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:09.154352 16319 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:09.175465 16171 leader_election.cc:290] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:09.155158 16353 raft_consensus.cc:493] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:09.187196 13529 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40916 (request call id 728) took 1336 ms (client timeout 1610 ms). Trace:
I20260504 20:02:09.187750 16317 raft_consensus.cc:493] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.188930 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.244236 12745 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 3.
I20260504 20:02:09.176011 16285 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.191263 12748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1905) took 1596 ms. Trace:
I20260504 20:02:09.193414 16217 raft_consensus.cc:515] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.194846 12893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37318 (request call id 1875) took 3467 ms. Trace:
I20260504 20:02:09.195519 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 537) took 1190 ms. Trace:
I20260504 20:02:09.198539 13524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 724) took 2388 ms. Trace:
I20260504 20:02:09.199297 16204 leader_election.cc:290] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.200781 16114 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.201547 13530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.201795 13528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 516) took 1266 ms. Trace:
I20260504 20:02:09.204598 12895 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37334 (request call id 2130) took 1214 ms. Trace:
I20260504 20:02:09.247036 12748 rpcz_store.cc:276] 0504 20:02:07.594655 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.595561 (+ 906us) service_pool.cc:224] Handling call
0504 20:02:07.595950 (+ 389us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:07.912997 (+317047us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:07.913044 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:07.913740 (+ 696us) log.cc:844] Serialized 26 byte log entry
0504 20:02:07.962413 (+ 48673us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:02:07.962468 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.190790 (+1228322us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b5400028854. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.190845 (+ 55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:09.190910 (+ 65us) raft_consensus.cc:1685] finished
0504 20:02:09.190924 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.191186 (+ 262us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10377,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":9603,"spinlock_wait_cycles":728105216,"thread_start_us":48012,"threads_started":1,"wal-append.queue_time_us":603135}
I20260504 20:02:09.252045 12895 rpcz_store.cc:276] 0504 20:02:07.990469 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.990701 (+ 232us) service_pool.cc:224] Handling call
0504 20:02:09.204378 (+1213677us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54002cbad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:09.204515 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413484928}
I20260504 20:02:09.148705 16279 raft_consensus.cc:493] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:09.208995 12897 raft_consensus.cc:1240] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 5. Current term is 7. Ops: []
I20260504 20:02:09.251701 13528 rpcz_store.cc:276] 0504 20:02:07.935468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.935709 (+ 241us) service_pool.cc:224] Handling call
0504 20:02:08.726979 (+791270us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:08.766956 (+ 39977us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.201475 (+434519us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b54005be350. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:09.201676 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1804572544}
I20260504 20:02:09.182716 12896 raft_consensus.cc:2468] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 5.
I20260504 20:02:09.102869 15951 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.219633 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.259590 12824 raft_consensus.cc:2468] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 8.
I20260504 20:02:09.164348 16235 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.226630 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.262147 12714 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.262519 12715 leader_election.cc:304] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:09.227092 12741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 531) took 1263 ms. Trace:
W20260504 20:02:09.185309 12825 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 2233) took 2506 ms (client timeout 1892 ms). Trace:
I20260504 20:02:09.222352 16362 leader_election.cc:290] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:09.240916 13529 rpcz_store.cc:269] 0504 20:02:07.850615 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.850830 (+ 215us) service_pool.cc:224] Handling call
0504 20:02:09.186945 (+1336115us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b540051d1d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:09.187087 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1751030528}
I20260504 20:02:09.247036 12791 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:09.165406 16322 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.240303 16003 raft_consensus.cc:493] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.269321 12817 raft_consensus.cc:2393] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 11: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:09.269428 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.271410 12748 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:09.271636 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.249051 12821 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 10.
I20260504 20:02:09.208201 16275 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.272207 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "445dd93bcaf04bae88b84d4eb9eddb60" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.249902 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.169464 16250 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:09.250922 12893 rpcz_store.cc:276] 0504 20:02:05.727158 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:05.868653 (+141495us) service_pool.cc:224] Handling call
0504 20:02:05.869008 (+ 355us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:07.923918 (+2054910us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.194518 (+1270600us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b5400208ad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.194749 (+ 231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4526821120}
I20260504 20:02:09.251011 12744 rpcz_store.cc:276] 0504 20:02:08.004724 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:08.004859 (+ 135us) service_pool.cc:224] Handling call
0504 20:02:08.005165 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:08.496212 (+491047us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:08.496261 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:08.496328 (+ 67us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:02:08.496388 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.195205 (+698817us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b54002bd9d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.195269 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.195441 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":493696,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":6228,"spinlock_wait_cycles":6234496,"thread_start_us":487866,"threads_started":1}
I20260504 20:02:09.275744 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.276484 12744 raft_consensus.cc:2468] T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 6.
I20260504 20:02:09.147348 16341 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:09.253037 13524 rpcz_store.cc:276] 0504 20:02:06.809982 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.810162 (+ 180us) service_pool.cc:224] Handling call
0504 20:02:06.816492 (+ 6330us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:08.741522 (+1925030us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.198257 (+456735us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b54005fafd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.198453 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4365330304}
W20260504 20:02:09.255468 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 261915us
I20260504 20:02:09.197731 16357 raft_consensus.cc:2749] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151) with higher term. Message: Invalid argument: T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 4. Current term is 5.
W20260504 20:02:09.263355 12825 rpcz_store.cc:269] 0504 20:02:06.678648 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.678798 (+ 150us) service_pool.cc:224] Handling call
0504 20:02:09.184995 (+2506197us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54003acfd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:09.185220 (+ 225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4173277952}
I20260504 20:02:09.263270 12741 rpcz_store.cc:276] 0504 20:02:07.964212 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.964393 (+ 181us) service_pool.cc:224] Handling call
0504 20:02:08.034569 (+ 70176us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:08.454283 (+419714us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:08.454318 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:08.454371 (+ 53us) raft_consensus.cc:1645] Marking committed up to 7
0504 20:02:08.454401 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.226701 (+772300us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54001ab654. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.226779 (+ 78us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.226995 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":964614400}
W20260504 20:02:09.239086 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 55583.6us
I20260504 20:02:09.268112 12750 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 10.
I20260504 20:02:09.269505 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.226097 16321 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.244254 16353 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:09.273196 12745 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:09.273859 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.305665 12821 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 9.
I20260504 20:02:09.298697 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.248090 16318 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.307204 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.307751 12742 raft_consensus.cc:2393] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 8: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:09.308141 12825 raft_consensus.cc:2468] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 3.
I20260504 20:02:09.308562 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.309145 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:09.308625 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.278028 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.278666 12791 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:09.310283 12821 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:02:09.310391 12744 raft_consensus.cc:2468] T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:09.311394 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.311465 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.307430 12714 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:02:09.312954 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:09.313514 12715 leader_election.cc:304] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:09.314147 13528 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 11.
I20260504 20:02:09.211625 16314 raft_consensus.cc:515] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.321462 12824 raft_consensus.cc:2410] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260504 20:02:09.323371 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.322149 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.316313 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.323617 12863 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:09.324550 12748 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:09.270349 16204 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:09.330302 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:09.335914 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.270202 16171 raft_consensus.cc:2804] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader election won for term 10
I20260504 20:02:09.324541 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85ec7d979c6d41db891f52c7e0bbcefe" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.356097 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.357321 12744 raft_consensus.cc:2468] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:09.358762 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.360369 12744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.316499 12790 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:09.348162 12715 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:09.322424 13499 leader_election.cc:304] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:09.318222 12714 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.277009 16278 raft_consensus.cc:2749] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260504 20:02:09.362963 12824 raft_consensus.cc:2393] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:09.365841 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.367063 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.367614 12824 raft_consensus.cc:2468] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:09.368510 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.307176 16358 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.256178 16285 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.369329 12817 raft_consensus.cc:2410] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260504 20:02:09.302702 16327 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:09.307798 16223 raft_consensus.cc:493] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:02:09.370810 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 47280.5us
I20260504 20:02:09.376827 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.380656 12741 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
W20260504 20:02:09.364894 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 46809.7us
I20260504 20:02:09.383458 12714 leader_election.cc:304] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:09.383549 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.384268 13526 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:02:09.384392 13525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00b8418b3a8441ee845a0591a75a9f17" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.377671 16267 raft_consensus.cc:2804] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:09.256834 16270 consensus_queue.cc:1048] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260504 20:02:09.270705 16279 raft_consensus.cc:515] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.325795 16357 raft_consensus.cc:2749] T dfa79aedaea6499facdfc14a3e7467f8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:09.377430 16283 raft_consensus.cc:2749] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:02:09.371373 12790 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 54979us
I20260504 20:02:09.372574 12791 leader_election.cc:304] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.322425 16362 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260504 20:02:09.375229 16303 consensus_meta.cc:287] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.650s user 0.010s sys 0.006s
I20260504 20:02:09.316102 16034 raft_consensus.cc:2749] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260504 20:02:09.376255 12821 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 10.
I20260504 20:02:09.267120 16261 consensus_queue.cc:1059] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.086s
I20260504 20:02:09.419027 16261 raft_consensus.cc:3055] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Stepping down as leader of term 9
I20260504 20:02:09.419255 16309 leader_election.cc:290] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:09.419485 16261 raft_consensus.cc:740] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:09.259752 16114 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:02:09.394094 12714 leader_election.cc:304] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:09.421118 12746 raft_consensus.cc:2468] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:09.363454 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.386696 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.422209 12745 raft_consensus.cc:2393] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 7: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:09.374923 16215 raft_consensus.cc:2804] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:02:09.386855 15834 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260504 20:02:09.396498 12715 leader_election.cc:304] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:09.353041 16204 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.438120 16038 raft_consensus.cc:2749] T 00b8418b3a8441ee845a0591a75a9f17 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:09.438493 16204 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
W20260504 20:02:09.439245 15605 consensus_meta.cc:287] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.739s user 0.015s sys 0.000s
W20260504 20:02:09.368636 16301 consensus_meta.cc:287] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.872s user 0.011s sys 0.004s
I20260504 20:02:09.250469 16317 raft_consensus.cc:515] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.403173 12864 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:09.299811 16322 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.275470 16235 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.300122 16003 raft_consensus.cc:515] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.407897 12791 leader_election.cc:304] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.314944 16321 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:02:09.307152 16250 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.427428 16215 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.452118 16215 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:09.453311 16250 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:09.378481 16282 consensus_queue.cc:1048] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
W20260504 20:02:09.415647 16367 log.cc:927] Time spent T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260504 20:02:09.418150 16305 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.433701 16339 raft_consensus.cc:2749] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:02:09.456460 12825 raft_consensus.cc:2468] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 11.
I20260504 20:02:09.395347 12750 raft_consensus.cc:2468] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:09.378732 16153 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260504 20:02:09.426137 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.376917 16171 raft_consensus.cc:697] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:09.476747 16171 consensus_queue.cc:237] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.467557 16114 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:09.385610 15951 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.381024 12863 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:09.385679 16267 raft_consensus.cc:493] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5ff0ef72ec3b4667893a0420d3c7f923)
I20260504 20:02:09.480986 16267 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:09.480698 16319 raft_consensus.cc:515] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.482509 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2235) took 1818 ms. Trace:
I20260504 20:02:09.482632 16279 leader_election.cc:290] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:09.483078 12820 rpcz_store.cc:276] 0504 20:02:07.664446 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.665759 (+ 1313us) service_pool.cc:224] Handling call
0504 20:02:07.666110 (+ 351us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:07.987287 (+321177us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:07.987325 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:07.987841 (+ 516us) log.cc:844] Serialized 26 byte log entry
0504 20:02:08.702894 (+715053us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:02:08.702943 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.482005 (+779062us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b5400374954. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.482066 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:09.482123 (+ 57us) raft_consensus.cc:1685] finished
0504 20:02:09.482138 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.482421 (+ 283us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21834,"raft.queue_time_us":941,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":34029,"spinlock_wait_cycles":737237760,"thread_start_us":462483,"threads_started":1,"wal-append.queue_time_us":714457}
I20260504 20:02:09.388048 16275 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.436210 12748 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 7.
I20260504 20:02:09.511180 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.398792 16223 raft_consensus.cc:515] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.386889 16217 leader_election.cc:290] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.404464 16182 raft_consensus.cc:2749] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 20:02:09.403195 16362 raft_consensus.cc:493] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.364708 16189 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.489560 16003 leader_election.cc:290] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.523320 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.369660 16306 consensus_queue.cc:1059] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 10, Time since last communication: 0.050s
I20260504 20:02:09.458833 12818 raft_consensus.cc:1275] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260504 20:02:09.525570 12818 pending_rounds.cc:85] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting all ops after (but not including) 4
W20260504 20:02:09.459699 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 51952.5us
I20260504 20:02:09.525877 12818 pending_rounds.cc:107] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260504 20:02:09.455156 16305 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.462721 12741 raft_consensus.cc:2410] T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260504 20:02:09.463140 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f8ca9e82e464c93b50b298fc802e0d0" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.424391 16278 raft_consensus.cc:2804] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260504 20:02:09.406683 16358 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.467490 16317 leader_election.cc:290] T 04d76f164cad4ed0bc8e39ce2403623f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.471364 16153 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.473187 16322 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.462476 16283 raft_consensus.cc:2804] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:02:09.354215 16318 consensus_queue.cc:1059] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260504 20:02:09.486135 16352 raft_consensus.cc:2804] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:09.486083 16261 consensus_queue.cc:260] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.486994 16321 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007)
I20260504 20:02:09.449491 16326 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:09.494939 16350 consensus_meta.cc:287] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.519s user 0.009s sys 0.004s
I20260504 20:02:09.497354 12716 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:09.422286 16303 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.498862 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.501947 12892 raft_consensus.cc:2368] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5.
I20260504 20:02:09.503551 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43dc05259beb4569a29aeb5327c9b3bc" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.506294 12750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.502058 12824 raft_consensus.cc:2468] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 9.
I20260504 20:02:09.501376 16235 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:09.508993 13499 leader_election.cc:304] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:09.436172 15834 raft_consensus.cc:493] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
W20260504 20:02:09.580989 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 83764.8us
I20260504 20:02:09.520323 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.417626 16285 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.523334 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.524963 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57454 (request call id 1948) took 2946 ms. Trace:
I20260504 20:02:09.526237 12818 raft_consensus.cc:2889] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 20:02:09.531366 12791 leader_election.cc:304] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.593443 12743 rpcz_store.cc:276] 0504 20:02:06.578871 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.582454 (+ 3583us) service_pool.cc:224] Handling call
0504 20:02:06.586706 (+ 4252us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:08.320474 (+1733768us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.524656 (+1204182us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540022fbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.524881 (+ 225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2770952960}
I20260504 20:02:09.443697 15605 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.536773 16223 leader_election.cc:290] T dfa79aedaea6499facdfc14a3e7467f8 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:09.536939 16374 log.cc:927] Time spent T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260504 20:02:09.539587 16305 leader_election.cc:290] T 1ebe61f64fd74c158829e717ae37a5b9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.599979 16235 raft_consensus.cc:2804] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 20:02:09.541185 12890 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.600471 16235 raft_consensus.cc:493] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.600891 16235 raft_consensus.cc:3060] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 4
I20260504 20:02:09.542411 16189 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:09.543076 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:09.532881 16275 consensus_queue.cc:1059] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260504 20:02:09.602442 16275 raft_consensus.cc:3055] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:02:09.602927 16275 raft_consensus.cc:740] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:09.603468 12742 raft_consensus.cc:1240] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 4. Current term is 5. Ops: []
I20260504 20:02:09.604097 16275 consensus_queue.cc:260] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.563369 12745 raft_consensus.cc:1240] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260504 20:02:09.607628 16303 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 10 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.559345 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37778afa14f24e82ade2bdc5c14b8a10" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:02:09.548997 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:09.562615 12741 raft_consensus.cc:1275] T c03254a421c44bdf89f865e637b18b2a P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260504 20:02:09.610610 12743 raft_consensus.cc:1240] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260504 20:02:09.530289 16278 raft_consensus.cc:493] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.555792 16353 raft_consensus.cc:515] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.614042 16278 raft_consensus.cc:3060] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:09.616628 16353 leader_election.cc:290] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.617650 12749 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: []
I20260504 20:02:09.618685 12741 raft_consensus.cc:1275] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260504 20:02:09.609508 16275 raft_consensus.cc:3060] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:09.617419 16204 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.582244 16352 raft_consensus.cc:493] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.621865 16352 raft_consensus.cc:3060] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:09.622259 12741 raft_consensus.cc:1240] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260504 20:02:09.516701 12899 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.584511 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:02:09.593961 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 62697.7us
I20260504 20:02:09.632476 12791 leader_election.cc:304] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.520406 16362 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:09.548041 16306 consensus_queue.cc:1243] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Peer 82787d8e64d24ca885ca517aa8ed1be2 log is divergent from this leader: its last log entry 9.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260504 20:02:09.598136 12750 raft_consensus.cc:1275] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260504 20:02:09.529994 16152 raft_consensus.cc:515] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.596570 16149 raft_consensus.cc:2804] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:02:09.592957 15834 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:09.585536 16350 raft_consensus.cc:515] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.609210 15605 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:09.548151 16153 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Advancing to term 12
I20260504 20:02:09.548532 16338 leader_election.cc:290] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:09.612172 12742 raft_consensus.cc:1240] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260504 20:02:09.612577 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.722426 12825 raft_consensus.cc:2468] T 6d6a540765d9427aa634550381fa5eb2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:02:09.723675 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.724442 12825 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 7.
I20260504 20:02:09.725134 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.623646 16204 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.626428 12747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57934 (request call id 535) took 1623 ms. Trace:
W20260504 20:02:09.586885 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 77990.1us
I20260504 20:02:09.627821 12822 raft_consensus.cc:1240] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:09.695197 16162 raft_consensus.cc:2804] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:09.727893 12822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2247) took 1889 ms. Trace:
I20260504 20:02:09.727159 12747 rpcz_store.cc:276] 0504 20:02:08.002699 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:08.002949 (+ 250us) service_pool.cc:224] Handling call
0504 20:02:08.009439 (+ 6490us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:08.578153 (+568714us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:08.578203 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:08.578271 (+ 68us) raft_consensus.cc:1645] Marking committed up to 6
0504 20:02:08.578320 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.626155 (+1047835us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b5400012554. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.626209 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.626358 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1307426176}
I20260504 20:02:09.728354 12822 rpcz_store.cc:276] 0504 20:02:07.855142 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:08.511476 (+656334us) service_pool.cc:224] Handling call
0504 20:02:09.627433 (+1115957us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b5800003698. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015974e941 00007f01598d36c1 00007f01598d3572 00007f01627901d9 00007f016190b5e0 00007f01619d476b 00007f01619d47f1 00007f0161996680 00007f0161980fe9 00007f0161980a92 00007f015aeead34
0504 20:02:09.627635 (+ 202us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:09.727580 (+ 99945us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.727826 (+ 246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9412864}
I20260504 20:02:09.629992 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f39e53fa82443a2ba4847a0750ee84c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.641623 12714 leader_election.cc:304] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:09.729282 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.636901 12791 leader_election.cc:304] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.585829 16003 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:09.558328 16319 leader_election.cc:290] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:09.589527 16358 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.591847 16283 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:09.704619 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:09.590600 16326 raft_consensus.cc:3055] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:02:09.594432 16282 raft_consensus.cc:493] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.747169 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.712389 12894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.630337 16261 raft_consensus.cc:3060] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:09.729771 12817 raft_consensus.cc:2468] T 7f39e53fa82443a2ba4847a0750ee84c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 10.
I20260504 20:02:09.751492 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.730221 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 2264) took 1009 ms. Trace:
I20260504 20:02:09.752470 12821 rpcz_store.cc:276] 0504 20:02:08.720568 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.728985 (+1008417us) service_pool.cc:224] Handling call
0504 20:02:09.730158 (+ 1173us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:02:09.730284 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 93509.1us
I20260504 20:02:09.753561 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.731318 12819 raft_consensus.cc:3060] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:09.708112 16149 raft_consensus.cc:493] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:09.732798 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 91321.5us
I20260504 20:02:09.759661 12714 leader_election.cc:304] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.733879 16306 raft_consensus.cc:2804] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:09.735113 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.732609 16162 raft_consensus.cc:493] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.714112 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa35f0107b5c4a97b13a1057c9b2bca2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.748279 12822 raft_consensus.cc:2468] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:02:09.749063 12891 raft_consensus.cc:1240] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260504 20:02:09.756531 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054fe686af134ff7b72d53edb9fdb32c" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:09.752122 12817 raft_consensus.cc:2468] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 4.
I20260504 20:02:09.767520 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:09.768414 12825 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 3.
W20260504 20:02:09.720624 16236 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.556s user 0.008s sys 0.004s
I20260504 20:02:09.778337 16236 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.766412 16162 raft_consensus.cc:3060] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:09.783145 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:09.783629 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:09.734141 16182 raft_consensus.cc:2749] T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:02:09.786460 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17c7354d697140a58c517d98642057ed" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.785439 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.744751 16003 raft_consensus.cc:697] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:09.754957 12791 leader_election.cc:304] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:02:09.804656 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 49773.3us
I20260504 20:02:09.804818 16304 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:09.746438 16283 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:09.805584 16304 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.783120 16282 raft_consensus.cc:515] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.806332 16304 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:09.763309 16204 raft_consensus.cc:493] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:09.765833 16306 raft_consensus.cc:493] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.809846 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2250) took 1887 ms. Trace:
I20260504 20:02:09.783383 12714 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.810281 12816 rpcz_store.cc:276] 0504 20:02:07.922227 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:08.577195 (+654968us) service_pool.cc:224] Handling call
0504 20:02:08.577576 (+ 381us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:08.706814 (+129238us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:08.706846 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:08.706903 (+ 57us) raft_consensus.cc:1645] Marking committed up to 5
0504 20:02:08.706946 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.809577 (+1102631us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b5400399c54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.809625 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.809754 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296588032}
I20260504 20:02:09.811733 12714 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.734272 13499 leader_election.cc:304] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:09.768128 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47a6e494e974412e9a7112344021d967" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.794301 16358 raft_consensus.cc:2804] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:09.781471 16236 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:09.796486 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83c11e4ac731424b899e837119487d11" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.798682 16271 raft_consensus.cc:3060] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:09.763798 16149 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:09.747645 16326 raft_consensus.cc:740] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:09.772164 16350 leader_election.cc:290] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:09.813356 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 79155.7us
I20260504 20:02:09.814635 12816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c03254a421c44bdf89f865e637b18b2a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.814903 12898 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.824936 12898 raft_consensus.cc:2468] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
I20260504 20:02:09.815440 12714 leader_election.cc:304] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.813591 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00885c8870d4ef98c5c0f9299c458ea" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:09.816442 12819 raft_consensus.cc:1275] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260504 20:02:09.814289 16318 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260504 20:02:09.829332 16318 raft_consensus.cc:493] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:09.829922 16318 raft_consensus.cc:3060] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:09.830245 12816 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2251) took 1907 ms. Trace:
I20260504 20:02:09.830747 12816 rpcz_store.cc:276] 0504 20:02:07.922721 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.829196 (+1906475us) service_pool.cc:224] Handling call
0504 20:02:09.829599 (+ 403us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:09.829832 (+ 233us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:09.829859 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:09.829913 (+ 54us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:02:09.829950 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.830019 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.830171 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 20:02:09.813488 16310 consensus_meta.cc:287] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.831s user 0.009s sys 0.006s
I20260504 20:02:09.826900 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 2229) took 3642 ms. Trace:
I20260504 20:02:09.822325 16321 consensus_queue.cc:1059] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260504 20:02:09.827492 16270 consensus_queue.cc:1059] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260504 20:02:09.814684 16358 raft_consensus.cc:493] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.814572 16306 raft_consensus.cc:3060] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:09.819487 16182 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260504 20:02:09.834429 16286 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260504 20:02:09.814175 16305 consensus_queue.cc:1059] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260504 20:02:09.835125 12819 rpcz_store.cc:276] 0504 20:02:06.184600 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:06.184774 (+ 174us) service_pool.cc:224] Handling call
0504 20:02:06.198130 (+ 13356us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:06.993166 (+795036us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b5400313ba0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9261a0 00007f015c9b05a0 00007f015c9b1611 00007f015c9ae16f 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c
0504 20:02:09.719382 (+2726216us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b580054bfd0. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c90a48a 00007f015c98c827 00007f015c9a53c7 00007f015c9a6bbe 00007f015c9b0f29 00007f015c9b17f6 00007f015c9ae16f 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34
0504 20:02:09.826550 (+107168us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.826832 (+ 282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3337487616}
I20260504 20:02:09.838932 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2238) took 2108 ms. Trace:
I20260504 20:02:09.833611 12791 leader_election.cc:304] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:09.841305 13528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4472f9104c524735bcfeed8474933a75" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.820832 16147 raft_consensus.cc:2804] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:02:09.849921 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:09.860498 13524 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:09.852787 12714 leader_election.cc:304] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:09.821943 16322 consensus_queue.cc:1048] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260504 20:02:09.854815 16182 raft_consensus.cc:493] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:09.855598 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 474655us
I20260504 20:02:09.832952 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2253) took 1902 ms. Trace:
I20260504 20:02:09.859961 12818 rpcz_store.cc:276] 0504 20:02:07.730314 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.730482 (+ 168us) service_pool.cc:224] Handling call
0504 20:02:07.730955 (+ 473us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:09.593757 (+1862802us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.838634 (+244877us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54003ae154. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:09.838854 (+ 220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3972905856}
I20260504 20:02:09.850546 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfa79aedaea6499facdfc14a3e7467f8" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:09.866837 12863 leader_election.cc:304] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:09.812220 16267 raft_consensus.cc:515] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.865074 12865 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:09.866715 12817 rpcz_store.cc:276] 0504 20:02:07.930338 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.832157 (+1901819us) service_pool.cc:224] Handling call
0504 20:02:09.832469 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:09.832658 (+ 189us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:09.832690 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0504 20:02:09.832720 (+ 30us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:02:09.832741 (+ 21us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.832780 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:09.832862 (+ 82us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:09.870607 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.864780 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.834321 16260 consensus_queue.cc:1048] T f00885c8870d4ef98c5c0f9299c458ea P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260504 20:02:09.851030 16358 raft_consensus.cc:3060] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:09.843564 16292 raft_consensus.cc:2804] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:09.874423 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.876156 13497 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:09.879196 12747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.890652 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
W20260504 20:02:09.884270 12791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:09.822551 16204 raft_consensus.cc:515] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } attrs { replace: true } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.855890 16321 consensus_queue.cc:1243] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer 82787d8e64d24ca885ca517aa8ed1be2 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260504 20:02:09.821149 16341 raft_consensus.cc:515] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.895141 16152 leader_election.cc:290] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:09.928881 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a80bf1f95a74ce09071a550fe15996f" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.845868 16310 raft_consensus.cc:515] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:09.930210 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ebe61f64fd74c158829e717ae37a5b9" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.931434 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:09.932921 12745 raft_consensus.cc:1240] T 86ea905fd808496c9b7d84a57c496cb3 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260504 20:02:09.933753 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.935564 16303 raft_consensus.cc:3055] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Stepping down as leader of term 4
I20260504 20:02:09.936002 16303 raft_consensus.cc:740] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:09.936296 12748 raft_consensus.cc:1275] T a87aaa2ccb5340acb2654b93a91d22b5 P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Refusing update from remote peer 5ff0ef72ec3b4667893a0420d3c7f923: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260504 20:02:09.934334 13529 raft_consensus.cc:1240] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:09.936818 16303 consensus_queue.cc:260] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:09.869001 16182 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:09.927392 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.940369 12825 raft_consensus.cc:2410] T 17c7354d697140a58c517d98642057ed P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260504 20:02:09.864637 16178 raft_consensus.cc:2804] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:09.914095 12746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea6d3aca49e4b1c9545e2ddb5b8e55b" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:09.928467 12818 raft_consensus.cc:1240] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 5. Current term is 6. Ops: []
I20260504 20:02:09.965299 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 2241) took 1014 ms. Trace:
I20260504 20:02:09.965900 12818 rpcz_store.cc:276] 0504 20:02:08.950737 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.871448 (+920711us) service_pool.cc:224] Handling call
0504 20:02:09.928249 (+ 56801us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:09.965033 (+ 36784us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.965227 (+ 194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:09.897534 13497 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
I20260504 20:02:09.940783 12741 raft_consensus.cc:1240] T 37778afa14f24e82ade2bdc5c14b8a10 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260504 20:02:09.946707 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 79993.4us
I20260504 20:02:09.951042 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968" is_pre_election: true
I20260504 20:02:09.865724 16353 raft_consensus.cc:2749] T f00885c8870d4ef98c5c0f9299c458ea P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260504 20:02:09.973016 13524 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:09.962870 12819 raft_consensus.cc:1240] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 9. Current term is 10. Ops: []
I20260504 20:02:09.871687 16034 raft_consensus.cc:3055] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Stepping down as leader of term 9
I20260504 20:02:09.873126 16147 raft_consensus.cc:697] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:02:09.881956 16270 raft_consensus.cc:2804] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260504 20:02:09.987274 12863 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
W20260504 20:02:09.992938 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:09.991623 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 126650us
I20260504 20:02:09.989732 12748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:10.015394 12865 leader_election.cc:304] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:10.014187 13530 raft_consensus.cc:1240] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:09.928627 16267 leader_election.cc:290] T 0832607bf7dd4d93a376269f0dd3dca1 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:09.925961 16292 raft_consensus.cc:493] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:09.872072 16357 consensus_queue.cc:1048] T 410644ff9c0449188275a6d49ecc0c2d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.110s
I20260504 20:02:09.964459 16178 raft_consensus.cc:493] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 20:02:10.036144 12646 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:36606 (request call id 120) took 26235 ms (client timeout 15000 ms). Trace:
I20260504 20:02:09.964733 16338 raft_consensus.cc:2804] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260504 20:02:10.044095 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:10.037258 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91ff767ed7c24b97940a459421881055" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:09.999388 16303 raft_consensus.cc:3060] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:09.982771 16034 raft_consensus.cc:740] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
W20260504 20:02:10.048866 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 70681.7us
I20260504 20:02:10.036202 16292 raft_consensus.cc:3060] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:10.052644 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 65484us
I20260504 20:02:10.035012 16270 raft_consensus.cc:493] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:10.053414 16338 raft_consensus.cc:493] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:09.974888 16202 raft_consensus.cc:493] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:10.058398 12818 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:10.050683 16178 raft_consensus.cc:3060] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:09.870294 16326 consensus_queue.cc:260] T 0f8ca9e82e464c93b50b298fc802e0d0 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:10.046067 12646 rpcz_store.cc:269] 0504 20:01:43.800983 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:01:43.803856 (+ 2873us) service_pool.cc:224] Handling call
0504 20:02:10.036016 (+26232160us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 20:02:03.419344 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 20:02:03.436049 (+ 16705us) write_op.cc:432] Acquiring schema lock in shared mode
0504 20:02:03.436108 (+ 59us) write_op.cc:435] Acquired schema lock
0504 20:02:03.436127 (+ 19us) tablet.cc:662] Decoding operations
0504 20:02:03.511953 (+ 75826us) write_op.cc:620] Acquiring the partition lock for write op
0504 20:02:03.522615 (+ 10662us) write_op.cc:641] Partition lock acquired for write op
0504 20:02:03.522651 (+ 36us) tablet.cc:685] Acquiring locks for 10 operations
0504 20:02:03.554593 (+ 31942us) tablet.cc:701] Row locks acquired
0504 20:02:03.554623 (+ 30us) write_op.cc:260] PREPARE: finished
0504 20:02:03.554886 (+ 263us) write_op.cc:270] Start()
0504 20:02:03.555076 (+ 190us) write_op.cc:276] Timestamp: P: 1777924923554788 usec, L: 0
0504 20:02:03.555133 (+ 57us) op_driver.cc:348] REPLICATION: starting
0504 20:02:03.556030 (+ 897us) log.cc:844] Serialized 3924 byte log entry
0504 20:02:04.372280 (+816250us) op_driver.cc:464] REPLICATION: finished
0504 20:02:04.916014 (+543734us) write_op.cc:301] APPLY: starting
0504 20:02:04.925126 (+ 9112us) tablet.cc:1367] starting BulkCheckPresence
0504 20:02:04.967454 (+ 42328us) tablet.cc:1370] finished BulkCheckPresence
0504 20:02:04.978067 (+ 10613us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 20:02:04.997638 (+ 19571us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 20:02:04.998029 (+ 391us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0504 20:02:04.998097 (+ 68us) write_op.cc:312] APPLY: finished
0504 20:02:04.998491 (+ 394us) log.cc:844] Serialized 77 byte log entry
0504 20:02:04.998715 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:02:04.998936 (+ 221us) write_op.cc:454] Released schema lock
0504 20:02:04.999021 (+ 85us) write_op.cc:341] FINISH: Updating metrics
0504 20:02:04.999575 (+ 554us) write_op.cc:489] Releasing partition, row and schema locks
0504 20:02:04.999613 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":423519,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":500880,"apply.run_cpu_time_us":5137,"apply.run_wall_time_us":105396,"num_ops":10,"prepare.queue_time_us":664907,"prepare.run_cpu_time_us":14310,"prepare.run_wall_time_us":245447,"raft.queue_time_us":505278,"raft.run_cpu_time_us":324939,"raft.run_wall_time_us":537904,"replication_time_us":759827,"thread_start_us":1414680,"threads_started":4,"wal-append.queue_time_us":218121,"wal-append.run_cpu_time_us":13525,"wal-append.run_wall_time_us":2222964}]]}
I20260504 20:02:10.088979 12864 leader_election.cc:304] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:10.090265 12818 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 11.
I20260504 20:02:10.092163 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d083393cd9f34e0b9d827ec5efd931eb" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:10.093356 12818 raft_consensus.cc:2393] T d083393cd9f34e0b9d827ec5efd931eb P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:02:10.097524 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:10.073549 16338 raft_consensus.cc:3060] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Advancing to term 12
I20260504 20:02:10.060659 13524 raft_consensus.cc:3060] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Advancing to term 11
I20260504 20:02:10.013641 15943 raft_consensus.cc:2764] T aada4371a0e5404bb6b69e5a2f3b1164 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260504 20:02:10.048552 12865 leader_election.cc:304] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:10.021375 16086 raft_consensus.cc:2804] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
W20260504 20:02:09.981248 12693 catalog_manager.cc:5194] ChangeConfig:REMOVE_PEER RPC for tablet 0e8eba4dbb8d436f81062ca3bbf46788 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 5
I20260504 20:02:09.882161 16003 consensus_queue.cc:237] T 71d0fab1020d42788e3e7a0c70e27637 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.017740 16222 raft_consensus.cc:2804] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 10 FOLLOWER]: Leader election won for term 10
I20260504 20:02:10.047076 16310 leader_election.cc:290] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:10.112950 16170 raft_consensus.cc:2804] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:02:10.054545 16279 raft_consensus.cc:2804] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader election won for term 9
I20260504 20:02:10.129331 16279 raft_consensus.cc:697] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [term 9 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:10.130375 16279 consensus_queue.cc:237] T 86ea905fd808496c9b7d84a57c496cb3 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.059098 16331 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:10.133901 16147 consensus_queue.cc:237] T 0f8ca9e82e464c93b50b298fc802e0d0 P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:10.134536 16331 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:10.134963 16331 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:10.070672 16270 raft_consensus.cc:3060] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:10.102725 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:10.138000 12818 raft_consensus.cc:3060] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:10.062572 16278 raft_consensus.cc:515] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:10.139087 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 241646us
I20260504 20:02:10.139966 13497 leader_election.cc:304] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters: 732c3560a688494196d22a1a48f89968
W20260504 20:02:10.086994 16383 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.303s user 0.000s sys 0.002s
I20260504 20:02:10.043823 12819 raft_consensus.cc:1240] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 8. Current term is 9. Ops: []
I20260504 20:02:10.077004 16319 consensus_queue.cc:1059] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260504 20:02:10.126609 13526 raft_consensus.cc:1275] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Refusing update from remote peer 732c3560a688494196d22a1a48f89968: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260504 20:02:10.128582 16307 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:10.051762 16217 consensus_queue.cc:1059] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260504 20:02:10.102389 16169 consensus_queue.cc:1048] T 640695edb69f427ea7660c9522bb01ef P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.045s
I20260504 20:02:10.161281 12824 raft_consensus.cc:1240] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 10. Current term is 11. Ops: []
I20260504 20:02:10.164789 12819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53098 (request call id 2256) took 1041 ms. Trace:
I20260504 20:02:10.165472 12819 rpcz_store.cc:276] 0504 20:02:09.123703 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.977733 (+854030us) service_pool.cc:224] Handling call
0504 20:02:10.043525 (+ 65792us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:10.142979 (+ 99454us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:10.164688 (+ 21709us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:10.081534 16202 raft_consensus.cc:515] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.146896 16188 raft_consensus.cc:2749] T 85ec7d979c6d41db891f52c7e0bbcefe P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260504 20:02:10.142204 12693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 167463us
I20260504 20:02:10.147217 16317 raft_consensus.cc:2804] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260504 20:02:10.202687 12819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb27f3eaaf74a3f9028f57cd8d93c9d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:02:10.172995 16235 consensus_meta.cc:287] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.572s user 0.016s sys 0.000s
W20260504 20:02:10.175335 16352 consensus_meta.cc:287] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.553s user 0.011s sys 0.006s
I20260504 20:02:10.238005 13526 raft_consensus.cc:1275] T fd0471a7f3374ceab25bb10c891e18e8 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260504 20:02:10.244536 13524 raft_consensus.cc:2468] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 11.
W20260504 20:02:10.216492 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:10.246580 12715 leader_election.cc:304] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:10.198995 16152 consensus_queue.cc:1059] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260504 20:02:10.257334 16250 raft_consensus.cc:3055] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:10.205714 15834 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:10.258071 16250 raft_consensus.cc:740] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:10.259234 16282 leader_election.cc:290] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.260185 15834 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:10.259598 16250 consensus_queue.cc:260] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:10.217483 16317 raft_consensus.cc:493] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:10.262373 16317 raft_consensus.cc:3060] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:10.262310 16250 raft_consensus.cc:3060] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:10.267815 12747 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 11.
I20260504 20:02:10.270306 12743 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1920) took 1202 ms. Trace:
W20260504 20:02:10.229123 16384 log.cc:927] Time spent T fa35f0107b5c4a97b13a1057c9b2bca2 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.265s user 0.000s sys 0.001s
I20260504 20:02:10.282069 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:10.301759 12891 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:10.270859 12743 rpcz_store.cc:276] 0504 20:02:09.067706 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.611615 (+543909us) service_pool.cc:224] Handling call
0504 20:02:09.627636 (+ 16021us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:09.628030 (+ 394us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:09.628063 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:09.628622 (+ 559us) log.cc:844] Serialized 26 byte log entry
0504 20:02:09.697059 (+ 68437us) raft_consensus.cc:1645] Marking committed up to 4
0504 20:02:09.697108 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.697357 (+ 249us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:10.269986 (+572629us) raft_consensus.cc:1685] finished
0504 20:02:10.270025 (+ 39us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:10.270215 (+ 190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":977,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":165,"spinlock_wait_cycles":147840,"thread_start_us":67851,"threads_started":1,"wal-append.queue_time_us":120938}
I20260504 20:02:10.136085 16034 consensus_queue.cc:260] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.261204 15834 consensus_queue.cc:1048] T fd0471a7f3374ceab25bb10c891e18e8 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260504 20:02:10.157464 16307 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:10.214718 16386 log.cc:927] Time spent T 47a6e494e974412e9a7112344021d967 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
W20260504 20:02:10.308480 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:10.182463 16300 raft_consensus.cc:493] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:10.250062 16006 raft_consensus.cc:3055] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:10.267343 16310 raft_consensus.cc:2804] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 FOLLOWER]: Leader election won for term 11
I20260504 20:02:10.238420 16341 leader_election.cc:290] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:10.309497 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 260137us
I20260504 20:02:10.337795 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
W20260504 20:02:10.332947 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 350696us
I20260504 20:02:10.350531 12865 leader_election.cc:304] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
I20260504 20:02:10.239912 16235 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.353217 16396 raft_consensus.cc:2804] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:10.372913 16278 leader_election.cc:290] T 7e652e6edc2c4497b0f7f50dc79598f2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 11 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.373101 16396 raft_consensus.cc:493] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:10.373636 16396 raft_consensus.cc:3060] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:10.373906 16362 consensus_meta.cc:287] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.665s user 0.020s sys 0.002s
W20260504 20:02:10.353610 16389 log.cc:927] Time spent T 756858b25f1f4125bf837cb8379e3b02 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260504 20:02:10.375211 12750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57462 (request call id 1929) took 1153 ms. Trace:
I20260504 20:02:10.355660 16310 raft_consensus.cc:697] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:10.375571 12750 rpcz_store.cc:276] 0504 20:02:09.221940 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.700553 (+478613us) service_pool.cc:224] Handling call
0504 20:02:09.700955 (+ 402us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:09.701385 (+ 430us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:09.701432 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:09.702097 (+ 665us) log.cc:844] Serialized 26 byte log entry
0504 20:02:09.808580 (+106483us) raft_consensus.cc:1645] Marking committed up to 3
0504 20:02:09.808791 (+ 211us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:09.808864 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:10.374969 (+566105us) raft_consensus.cc:1685] finished
0504 20:02:10.374994 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:10.375139 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1157,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":4916,"thread_start_us":73554,"threads_started":1,"wal-append.queue_time_us":127729}
I20260504 20:02:10.376524 16235 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 4 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.376411 16310 consensus_queue.cc:237] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:10.357031 16215 consensus_meta.cc:287] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.904s user 0.006s sys 0.006s
I20260504 20:02:10.427979 12818 raft_consensus.cc:2468] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 8.
I20260504 20:02:10.429314 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aada4371a0e5404bb6b69e5a2f3b1164" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:10.430123 12818 raft_consensus.cc:3060] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 3 FOLLOWER]: Advancing to term 5
I20260504 20:02:10.358245 16358 raft_consensus.cc:515] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:10.434609 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 84256.6us
I20260504 20:02:10.360412 16307 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:10.362876 16357 consensus_queue.cc:1059] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 20:02:10.250658 16352 raft_consensus.cc:515] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.374414 16362 raft_consensus.cc:515] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.329720 16034 raft_consensus.cc:3060] T 054fe686af134ff7b72d53edb9fdb32c P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:10.398986 12791 leader_election.cc:304] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:10.436662 12865 leader_election.cc:304] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:02:10.457100 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 58241.6us
I20260504 20:02:10.330003 16006 raft_consensus.cc:740] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:10.457494 12865 leader_election.cc:304] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:10.458665 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:10.463677 16352 leader_election.cc:290] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.466820 16362 leader_election.cc:290] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:10.467499 16162 consensus_meta.cc:287] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.655s user 0.005s sys 0.009s
I20260504 20:02:10.468005 16162 raft_consensus.cc:515] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.468588 16278 raft_consensus.cc:2749] T 5409cb132bc14d1cafaa29b22ced9dc9 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:10.465544 12744 raft_consensus.cc:2393] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 10: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:10.473592 12791 leader_election.cc:304] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:10.475581 12821 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:10.446816 13526 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 11.
I20260504 20:02:10.452581 12637 maintenance_manager.cc:419] P 109800bd8ea34eaea0c7308025ecde32: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033339
I20260504 20:02:10.533322 12818 raft_consensus.cc:2468] T aada4371a0e5404bb6b69e5a2f3b1164 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 5.
I20260504 20:02:10.637948 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1271e7bfcea48b8bd072293da65c55d" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:10.481726 12715 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:10.639245 12742 raft_consensus.cc:1240] T 054fe686af134ff7b72d53edb9fdb32c P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260504 20:02:10.640174 12818 raft_consensus.cc:2393] T b1271e7bfcea48b8bd072293da65c55d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 6: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:02:10.640637 12715 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.4:41009, user_credentials={real_user=slave}} blocked reactor thread for 158863us
I20260504 20:02:10.641793 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:10.643002 12818 raft_consensus.cc:3060] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Advancing to term 10
W20260504 20:02:10.503297 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 159670us
I20260504 20:02:10.458966 12865 leader_election.cc:304] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 732c3560a688494196d22a1a48f89968, 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923
W20260504 20:02:10.652453 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 193413us
I20260504 20:02:10.652690 12863 leader_election.cc:304] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:10.532233 12891 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:02:10.480984 12820 raft_consensus.cc:2393] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:02:10.663036 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 202408us
W20260504 20:02:10.667652 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:10.507619 13524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:10.675786 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:10.676545 16034 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:10.677846 12742 raft_consensus.cc:1240] T bce60695d3db4c06ad8e0eec0419f79f P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 LEADER]: Rejecting Update request from peer 5ff0ef72ec3b4667893a0420d3c7f923 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260504 20:02:10.677644 16034 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.438349 16003 consensus_queue.cc:1059] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.089s
I20260504 20:02:10.544766 16006 consensus_queue.cc:260] T af25f180a18a4e79a93b35bb0a2e55f0 P 732c3560a688494196d22a1a48f89968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.674576 15834 raft_consensus.cc:2804] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 FOLLOWER]: Leader election won for term 11
I20260504 20:02:10.674870 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "640695edb69f427ea7660c9522bb01ef" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:10.494554 16086 raft_consensus.cc:697] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:02:10.561769 16149 consensus_meta.cc:287] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.727s user 0.009s sys 0.005s
I20260504 20:02:10.572198 16162 leader_election.cc:290] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.528220 16170 raft_consensus.cc:697] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:10.572317 16202 leader_election.cc:290] T 4318d305513d467c8254d7c054ef993e P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:10.459142 16204 leader_election.cc:290] T 0e8eba4dbb8d436f81062ca3bbf46788 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:10.482798 16208 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Advancing to term 8
I20260504 20:02:10.453647 16215 raft_consensus.cc:515] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.484750 16222 raft_consensus.cc:697] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [term 10 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:10.782116 12745 raft_consensus.cc:2468] T 640695edb69f427ea7660c9522bb01ef P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 8.
I20260504 20:02:10.452960 16235 consensus_queue.cc:1048] T c03254a421c44bdf89f865e637b18b2a P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20260504 20:02:10.783867 16286 raft_consensus.cc:493] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:10.784957 16286 raft_consensus.cc:515] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:10.494124 16278 raft_consensus.cc:2804] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260504 20:02:10.814977 16278 raft_consensus.cc:493] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:10.815689 16278 raft_consensus.cc:3060] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
I20260504 20:02:10.574616 16285 raft_consensus.cc:3055] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Stepping down as leader of term 5
I20260504 20:02:10.330916 16300 raft_consensus.cc:515] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.818279 16285 raft_consensus.cc:740] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:10.818339 12744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57462 (request call id 1919) took 1750 ms. Trace:
I20260504 20:02:10.480008 16307 raft_consensus.cc:2804] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election won for term 5
I20260504 20:02:10.819051 12744 rpcz_store.cc:276] 0504 20:02:09.067373 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.360035 (+292662us) service_pool.cc:224] Handling call
0504 20:02:10.817651 (+1457616us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400028d54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:10.818050 (+ 399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2539005824}
I20260504 20:02:10.576830 16314 leader_election.cc:290] T 756858b25f1f4125bf837cb8379e3b02 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:10.820734 16307 raft_consensus.cc:697] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [term 5 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
W20260504 20:02:10.562552 16318 consensus_meta.cc:287] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.732s user 0.011s sys 0.004s
W20260504 20:02:10.822511 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 111245us
I20260504 20:02:10.822623 16318 raft_consensus.cc:515] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.477312 16330 consensus_queue.cc:1059] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260504 20:02:10.529470 16331 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.581588 16399 consensus_queue.cc:1048] T fe81ef9fbbe44f89a19e923171496fa5 P 732c3560a688494196d22a1a48f89968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260504 20:02:10.670189 12865 leader_election.cc:304] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:10.824994 16314 raft_consensus.cc:3055] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 LEADER]: Stepping down as leader of term 8
I20260504 20:02:10.826301 16314 raft_consensus.cc:740] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:10.828329 16331 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:10.827945 16314 consensus_queue.cc:260] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.568439 16292 raft_consensus.cc:515] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } }
I20260504 20:02:10.829356 16331 raft_consensus.cc:3060] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Advancing to term 6
W20260504 20:02:10.729277 16304 consensus_meta.cc:287] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.922s user 0.011s sys 0.005s
I20260504 20:02:10.849890 12817 raft_consensus.cc:1240] T fa35f0107b5c4a97b13a1057c9b2bca2 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 6. Current term is 7. Ops: []
I20260504 20:02:10.850407 16304 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.734684 16358 leader_election.cc:290] T 6d6a540765d9427aa634550381fa5eb2 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.853783 16285 consensus_queue.cc:260] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.738077 16352 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:10.855384 16285 raft_consensus.cc:3060] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Advancing to term 6
I20260504 20:02:10.855628 16352 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:10.739367 16034 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.745519 16149 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:10.859678 16304 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.860813 16149 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.746538 16086 consensus_queue.cc:237] T 17c7354d697140a58c517d98642057ed P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:10.748694 16153 consensus_meta.cc:287] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.027s user 0.018s sys 0.002s
I20260504 20:02:10.719683 12791 leader_election.cc:304] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:10.748749 16397 raft_consensus.cc:2804] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader election won for term 8
I20260504 20:02:10.868978 16286 leader_election.cc:290] T 410644ff9c0449188275a6d49ecc0c2d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:10.869268 16397 raft_consensus.cc:697] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [term 8 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:10.873070 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe81ef9fbbe44f89a19e923171496fa5" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:10.873999 13526 raft_consensus.cc:2468] T fe81ef9fbbe44f89a19e923171496fa5 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 7.
W20260504 20:02:10.874249 16303 consensus_meta.cc:287] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.802s user 0.011s sys 0.004s
I20260504 20:02:10.719568 13499 leader_election.cc:304] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 732c3560a688494196d22a1a48f89968, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:10.875515 12819 raft_consensus.cc:2410] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260504 20:02:10.875707 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 205922us
W20260504 20:02:10.876252 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:10.876298 16300 leader_election.cc:290] T e67d58ddf5464b6daea5f094d46a05b6 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:10.869730 16286 raft_consensus.cc:2749] T 7a80bf1f95a74ce09071a550fe15996f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260504 20:02:10.879933 12790 leader_election.cc:304] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:10.783383 16222 consensus_queue.cc:237] T 7f39e53fa82443a2ba4847a0750ee84c P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.881235 16303 raft_consensus.cc:2804] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260504 20:02:10.881704 16303 raft_consensus.cc:493] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 20:02:10.882345 16303 raft_consensus.cc:3060] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:02:10.586055 16271 consensus_meta.cc:287] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.764s user 0.017s sys 0.002s
I20260504 20:02:10.885263 16143 raft_consensus.cc:2804] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 FOLLOWER]: Leader election won for term 7
I20260504 20:02:10.885432 12823 raft_consensus.cc:2468] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 6.
I20260504 20:02:10.886353 16144 raft_consensus.cc:493] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:10.772308 16170 consensus_queue.cc:237] T aada4371a0e5404bb6b69e5a2f3b1164 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:10.803821 12742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:10.807845 12745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "faf44d4cfb914599a82b50e6d195ecbd" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:10.809056 12741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "756858b25f1f4125bf837cb8379e3b02" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2" is_pre_election: true
I20260504 20:02:10.812983 16400 raft_consensus.cc:2804] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Leader election won for term 7
W20260504 20:02:10.813884 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 69436.9us
W20260504 20:02:10.830040 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:10.830147 16314 raft_consensus.cc:3060] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Advancing to term 9
W20260504 20:02:10.594069 16283 consensus_meta.cc:287] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.788s user 0.013s sys 0.003s
I20260504 20:02:10.822569 16307 consensus_queue.cc:237] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:10.845258 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 191460us
I20260504 20:02:10.846504 16318 leader_election.cc:290] T 7f39e53fa82443a2ba4847a0750ee84c P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:10.846985 12820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 2253) took 1724 ms (client timeout 2214 ms). Trace:
I20260504 20:02:10.848796 13530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40924 (request call id 522) took 1585 ms. Trace:
I20260504 20:02:10.851735 12817 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2269) took 1858 ms. Trace:
I20260504 20:02:10.746933 16250 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:10.747213 12749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e652e6edc2c4497b0f7f50dc79598f2" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "82787d8e64d24ca885ca517aa8ed1be2"
I20260504 20:02:10.861532 16352 leader_election.cc:290] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:10.866367 16215 leader_election.cc:290] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 4 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:10.866324 16153 raft_consensus.cc:515] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:10.868495 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 198975us
I20260504 20:02:10.888340 16397 consensus_queue.cc:237] T 431221deee84429b8261f87972c51218 P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.887017 16144 raft_consensus.cc:515] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.825412 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53102 (request call id 2265) took 2104 ms. Trace:
W20260504 20:02:10.923282 12790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:10.908463 13530 rpcz_store.cc:276] 0504 20:02:09.263040 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.263271 (+ 231us) service_pool.cc:224] Handling call
0504 20:02:09.325641 (+ 62370us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:10.025070 (+699429us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:10.848043 (+822973us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b540058c354. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:10.848504 (+ 461us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1582816512}
I20260504 20:02:10.944383 12742 raft_consensus.cc:2393] T 927c932a914241598bc58bcb0d84c2d4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 4: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
W20260504 20:02:10.907933 12820 rpcz_store.cc:269] 0504 20:02:09.122789 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.696852 (+574063us) service_pool.cc:224] Handling call
0504 20:02:10.846504 (+1149652us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b540015fc54. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:10.846738 (+ 234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648021376}
I20260504 20:02:10.909179 12817 rpcz_store.cc:276] 0504 20:02:08.993221 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.873327 (+880106us) service_pool.cc:224] Handling call
0504 20:02:10.849333 (+976006us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:10.850461 (+ 1128us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:10.851506 (+ 1045us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b540015fc50. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9adc8d 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:10.851663 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2220552576}
I20260504 20:02:10.952004 12818 raft_consensus.cc:2468] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 10.
I20260504 20:02:10.943735 12824 rpcz_store.cc:276] 0504 20:02:08.720894 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.850485 (+1129591us) service_pool.cc:224] Handling call
0504 20:02:09.850797 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:10.161756 (+310959us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:10.824839 (+663083us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b5400399754. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:10.825175 (+ 336us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":701729152}
I20260504 20:02:10.963179 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410644ff9c0449188275a6d49ecc0c2d" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" is_pre_election: true
I20260504 20:02:10.915992 15834 raft_consensus.cc:697] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [term 11 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:10.968195 12745 raft_consensus.cc:2393] T faf44d4cfb914599a82b50e6d195ecbd P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ff0ef72ec3b4667893a0420d3c7f923 in current term 9: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:10.967773 16292 leader_election.cc:290] T 2c03c00d6aaa4086b0726c5dfc8fd3ff P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 7 election: Requested vote from peers 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:10.986457 16261 consensus_meta.cc:287] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.033s user 0.016s sys 0.003s
I20260504 20:02:10.986106 16396 raft_consensus.cc:515] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.941754 16143 raft_consensus.cc:697] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 7 LEADER]: Becoming Leader. State: Replica: ef6057d9a49e4942b47ea0fcd66ab6f7, State: Running, Role: LEADER
I20260504 20:02:10.944082 16400 raft_consensus.cc:697] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [term 7 LEADER]: Becoming Leader. State: Replica: 732c3560a688494196d22a1a48f89968, State: Running, Role: LEADER
I20260504 20:02:10.950958 16250 raft_consensus.cc:515] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:10.996470 12891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
W20260504 20:02:11.178508 16407 log.cc:927] Time spent T 756858b25f1f4125bf837cb8379e3b02 P ef6057d9a49e4942b47ea0fcd66ab6f7: Append to log took a long time: real 0.178s user 0.000s sys 0.001s
I20260504 20:02:11.183893 16143 consensus_queue.cc:237] T f2114545cfec4b19a5a3d1d6751f73da P ef6057d9a49e4942b47ea0fcd66ab6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.148682 16300 raft_consensus.cc:493] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:11.158789 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 69184.5us
W20260504 20:02:11.076687 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:11.033995 16283 raft_consensus.cc:515] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.194505 12891 raft_consensus.cc:2393] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in current term 5: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:02:11.194696 16400 consensus_queue.cc:237] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 732c3560a688494196d22a1a48f89968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.196996 16283 leader_election.cc:290] T 57fe021572534bbdbbaea4aa4a64017a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:11.050045 12865 leader_election.cc:304] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
W20260504 20:02:11.199473 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 230913us
I20260504 20:02:11.085578 12714 leader_election.cc:304] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:11.207299 16413 raft_consensus.cc:2804] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 FOLLOWER]: Leader election won for term 10
I20260504 20:02:11.207437 16400 raft_consensus.cc:2749] T 47a6e494e974412e9a7112344021d967 P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260504 20:02:11.208083 16413 raft_consensus.cc:697] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [term 10 LEADER]: Becoming Leader. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
W20260504 20:02:11.056025 16182 consensus_meta.cc:287] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 1.066s user 0.017s sys 0.005s
I20260504 20:02:11.209335 16413 consensus_queue.cc:237] T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.210697 16182 raft_consensus.cc:515] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.212960 16182 leader_election.cc:290] T 431221deee84429b8261f87972c51218 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.007186 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04d76f164cad4ed0bc8e39ce2403623f" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:02:11.214069 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 128696us
I20260504 20:02:11.008944 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0832607bf7dd4d93a376269f0dd3dca1" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:10.999693 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d6a540765d9427aa634550381fa5eb2" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
W20260504 20:02:11.221022 16270 consensus_meta.cc:287] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.071s user 0.020s sys 0.003s
I20260504 20:02:11.222864 13526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7"
I20260504 20:02:11.223680 13526 raft_consensus.cc:3060] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Advancing to term 12
I20260504 20:02:11.228317 16208 raft_consensus.cc:2749] T 37778afa14f24e82ade2bdc5c14b8a10 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260504 20:02:11.229092 12820 raft_consensus.cc:2468] T 04d76f164cad4ed0bc8e39ce2403623f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 9.
I20260504 20:02:11.229446 16271 consensus_queue.cc:1059] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260504 20:02:11.230325 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "927c932a914241598bc58bcb0d84c2d4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.230513 16034 raft_consensus.cc:3055] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:11.230952 16034 raft_consensus.cc:740] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:11.231263 12820 raft_consensus.cc:2393] T 927c932a914241598bc58bcb0d84c2d4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 4: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:11.231962 16034 consensus_queue.cc:260] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.233083 12817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2114545cfec4b19a5a3d1d6751f73da" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.233482 16034 raft_consensus.cc:3060] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Advancing to term 7
W20260504 20:02:11.033207 16317 consensus_meta.cc:287] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 0.770s user 0.004s sys 0.022s
I20260504 20:02:11.234467 16317 raft_consensus.cc:515] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
W20260504 20:02:11.079862 16306 consensus_meta.cc:287] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 1.229s user 0.006s sys 0.016s
W20260504 20:02:11.060153 16178 consensus_meta.cc:287] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.957s user 0.019s sys 0.004s
I20260504 20:02:11.051401 16153 leader_election.cc:290] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 12 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:11.148072 16318 raft_consensus.cc:493] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:02:11.281648 16390 log.cc:927] Time spent T 47a6e494e974412e9a7112344021d967 P 5ff0ef72ec3b4667893a0420d3c7f923: Append to log took a long time: real 0.983s user 0.008s sys 0.001s
I20260504 20:02:11.148614 16286 consensus_queue.cc:1059] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.380s
I20260504 20:02:11.060114 16399 consensus_queue.cc:1059] T fa35f0107b5c4a97b13a1057c9b2bca2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.469s
I20260504 20:02:11.529237 16318 raft_consensus.cc:515] T 1ebe61f64fd74c158829e717ae37a5b9 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.529693 16304 raft_consensus.cc:3055] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Stepping down as leader of term 7
I20260504 20:02:11.493870 15605 raft_consensus.cc:493] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ef6057d9a49e4942b47ea0fcd66ab6f7)
I20260504 20:02:11.530404 16304 raft_consensus.cc:740] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:11.530319 16306 raft_consensus.cc:515] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.442839 16271 raft_consensus.cc:493] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:11.531111 16178 raft_consensus.cc:515] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.531837 12822 raft_consensus.cc:2393] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 9: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:11.531492 16271 raft_consensus.cc:515] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.531637 16304 consensus_queue.cc:260] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.251830 16317 leader_election.cc:290] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 10 election: Requested vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.238711 16396 leader_election.cc:290] T a87aaa2ccb5340acb2654b93a91d22b5 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:11.533248 16304 raft_consensus.cc:3060] T 37778afa14f24e82ade2bdc5c14b8a10 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Advancing to term 8
W20260504 20:02:11.282073 12863 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 66050.8us
I20260504 20:02:11.442971 16416 raft_consensus.cc:493] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:11.534605 16306 leader_election.cc:290] T 83c11e4ac731424b899e837119487d11 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.479005 16182 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 732c3560a688494196d22a1a48f89968)
W20260504 20:02:11.375957 16285 consensus_meta.cc:287] T b1271e7bfcea48b8bd072293da65c55d P 82787d8e64d24ca885ca517aa8ed1be2: Time spent flushing consensus metadata: real 0.520s user 0.014s sys 0.001s
I20260504 20:02:11.535171 16416 raft_consensus.cc:515] T 7ddedcb68670423f9dbf822898141e0f P 82787d8e64d24ca885ca517aa8ed1be2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.052407 16261 consensus_queue.cc:1048] T a87aaa2ccb5340acb2654b93a91d22b5 P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.484s
W20260504 20:02:11.387871 16331 consensus_meta.cc:287] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968: Time spent flushing consensus metadata: real 0.538s user 0.012s sys 0.000s
I20260504 20:02:11.386466 12816 raft_consensus.cc:1240] T faf44d4cfb914599a82b50e6d195ecbd P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
W20260504 20:02:11.241961 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 20:02:11.332351 12791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 90736.6us
I20260504 20:02:11.076992 16352 raft_consensus.cc:493] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 20:02:11.375765 16278 consensus_meta.cc:287] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.559s user 0.013s sys 0.002s
I20260504 20:02:11.496403 16144 leader_election.cc:290] T 7cd4fffdb6254bd49f3d6ef2f95406ce P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.221776 16270 raft_consensus.cc:515] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:11.333194 13499 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 49546.6us
W20260504 20:02:11.484592 13496 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:11.249104 13527 raft_consensus.cc:1240] T 593a993b4ff341c9b98629e2c051ce00 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 10 FOLLOWER]: Rejecting Update request from peer 732c3560a688494196d22a1a48f89968 for earlier term 9. Current term is 10. Ops: []
I20260504 20:02:11.518738 16393 raft_consensus.cc:3055] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Stepping down as leader of term 9
I20260504 20:02:11.282279 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4318d305513d467c8254d7c054ef993e" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:02:11.452158 12714 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:11.364002 12865 leader_election.cc:304] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:11.356390 16208 consensus_queue.cc:1059] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.185s
I20260504 20:02:11.034607 15834 consensus_queue.cc:237] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.381417 12821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 2276) took 1753 ms. Trace:
I20260504 20:02:11.439870 12824 raft_consensus.cc:2468] T 0832607bf7dd4d93a376269f0dd3dca1 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:02:11.380137 12825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 2247) took 2393 ms. Trace:
I20260504 20:02:11.533552 16178 leader_election.cc:290] T 71d0fab1020d42788e3e7a0c70e27637 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
W20260504 20:02:11.547241 12788 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:11.282647 16300 raft_consensus.cc:515] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.530822 15605 raft_consensus.cc:515] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.553586 16393 raft_consensus.cc:740] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:11.556195 16311 raft_consensus.cc:493] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:11.562668 12824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48774 (request call id 401) took 1024 ms. Trace:
I20260504 20:02:11.563154 12824 rpcz_store.cc:276] 0504 20:02:10.534281 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:10.997175 (+462894us) service_pool.cc:224] Handling call
0504 20:02:11.558425 (+561250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:11.548585 16331 raft_consensus.cc:2749] T d083393cd9f34e0b9d827ec5efd931eb P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260504 20:02:11.558652 12821 rpcz_store.cc:276] 0504 20:02:09.627508 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.767228 (+139720us) service_pool.cc:224] Handling call
0504 20:02:11.380957 (+1613729us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54003002d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:11.381228 (+ 271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612851072}
I20260504 20:02:11.557993 12825 rpcz_store.cc:276] 0504 20:02:08.986331 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.786110 (+799779us) service_pool.cc:224] Handling call
0504 20:02:11.379620 (+1593510us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b540015dbd4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:11.379906 (+ 286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":351145344}
I20260504 20:02:11.546124 16271 leader_election.cc:290] T ddb27f3eaaf74a3f9028f57cd8d93c9d P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.573764 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9a8aee69ec8499895f3ab719968d204" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:11.576155 12824 raft_consensus.cc:2468] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 732c3560a688494196d22a1a48f89968 in term 4.
I20260504 20:02:11.578933 12820 raft_consensus.cc:2393] T 4318d305513d467c8254d7c054ef993e P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:11.582695 13526 raft_consensus.cc:2468] T 1e115d71520b4beb9b2984dd586354e4 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in term 12.
I20260504 20:02:11.551446 16278 raft_consensus.cc:515] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
W20260504 20:02:11.587173 12716 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.3:39151, user_credentials={real_user=slave}} blocked reactor thread for 231906us
I20260504 20:02:11.596163 16278 leader_election.cc:290] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 7 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.573024 16182 raft_consensus.cc:515] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.597352 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e115d71520b4beb9b2984dd586354e4" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.597990 16311 raft_consensus.cc:515] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.598678 15605 leader_election.cc:290] T ae27dbfd0ab84c9d8c95896c0c4cf502 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
I20260504 20:02:11.599167 12863 leader_election.cc:304] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968; no voters:
I20260504 20:02:11.599529 16182 leader_election.cc:290] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.599812 16362 raft_consensus.cc:3055] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Stepping down as leader of term 8
W20260504 20:02:11.600836 12747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57454 (request call id 1969) took 1728 ms (client timeout 1567 ms). Trace:
I20260504 20:02:11.603493 12818 raft_consensus.cc:2368] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 11. Current term is 12.
I20260504 20:02:11.597402 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36b69b3d13384774a38b1442833e6948" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.604290 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71d0fab1020d42788e3e7a0c70e27637" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.553094 16352 raft_consensus.cc:515] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.612118 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 2281) took 1255 ms. Trace:
I20260504 20:02:11.577970 16285 raft_consensus.cc:493] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:11.614946 12897 raft_consensus.cc:1275] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [term 5 FOLLOWER]: Refusing update from remote peer ef6057d9a49e4942b47ea0fcd66ab6f7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260504 20:02:11.612812 12824 raft_consensus.cc:2393] T 71d0fab1020d42788e3e7a0c70e27637 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 8: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
W20260504 20:02:11.617673 12716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:11.618064 12820 raft_consensus.cc:2393] T 36b69b3d13384774a38b1442833e6948 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 10: Already voted for candidate 82787d8e64d24ca885ca517aa8ed1be2 in this term.
I20260504 20:02:11.618607 12818 rpcz_store.cc:276] 0504 20:02:10.356291 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:11.596916 (+1240625us) service_pool.cc:224] Handling call
0504 20:02:11.612047 (+ 15131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37632}
I20260504 20:02:11.608865 12821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c03c00d6aaa4086b0726c5dfc8fd3ff" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.591293 16208 consensus_queue.cc:1243] T 7e652e6edc2c4497b0f7f50dc79598f2 P 732c3560a688494196d22a1a48f89968 [LEADER]: Peer 5ff0ef72ec3b4667893a0420d3c7f923 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260504 20:02:11.609730 16303 consensus_meta.cc:287] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.724s user 0.014s sys 0.001s
W20260504 20:02:11.611797 12747 rpcz_store.cc:269] 0504 20:02:09.872572 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.872766 (+ 194us) service_pool.cc:224] Handling call
0504 20:02:11.600462 (+1727696us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54002e4ad4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:11.600632 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":866567808}
I20260504 20:02:11.622398 12825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0471a7f3374ceab25bb10c891e18e8" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:11.626709 12820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 2281) took 1515 ms. Trace:
I20260504 20:02:11.629088 12820 rpcz_store.cc:276] 0504 20:02:10.106948 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:11.596946 (+1489998us) service_pool.cc:224] Handling call
0504 20:02:11.622017 (+ 25071us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3968}
I20260504 20:02:11.633062 16273 raft_consensus.cc:3055] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Stepping down as leader of term 6
I20260504 20:02:11.633697 16273 raft_consensus.cc:740] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5ff0ef72ec3b4667893a0420d3c7f923, State: Running, Role: LEADER
I20260504 20:02:11.633177 12714 leader_election.cc:304] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82787d8e64d24ca885ca517aa8ed1be2; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 732c3560a688494196d22a1a48f89968
I20260504 20:02:11.634888 16392 raft_consensus.cc:2749] T 17c7354d697140a58c517d98642057ed P 82787d8e64d24ca885ca517aa8ed1be2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:11.637830 12714 leader_election.cc:400] T 1e115d71520b4beb9b2984dd586354e4 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 11 election: Vote denied by peer 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415) with higher term. Message: Invalid argument: T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 11. Current term is 12.
I20260504 20:02:11.638752 12825 raft_consensus.cc:2410] T fd0471a7f3374ceab25bb10c891e18e8 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef6057d9a49e4942b47ea0fcd66ab6f7 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260504 20:02:11.639094 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cd4fffdb6254bd49f3d6ef2f95406ce" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:11.609305 16250 leader_election.cc:290] T c9a8aee69ec8499895f3ab719968d204 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.611657 16362 raft_consensus.cc:740] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 82787d8e64d24ca885ca517aa8ed1be2, State: Running, Role: LEADER
I20260504 20:02:11.643947 12791 leader_election.cc:304] T 640695edb69f427ea7660c9522bb01ef P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
W20260504 20:02:11.651420 12865 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.1:44007, user_credentials={real_user=slave}} blocked reactor thread for 396835us
I20260504 20:02:11.638025 12895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "593a993b4ff341c9b98629e2c051ce00" candidate_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "732c3560a688494196d22a1a48f89968"
I20260504 20:02:11.658066 16189 raft_consensus.cc:2749] T 054fe686af134ff7b72d53edb9fdb32c P 732c3560a688494196d22a1a48f89968 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260504 20:02:11.659361 16423 log.cc:927] Time spent T 36b69b3d13384774a38b1442833e6948 P 82787d8e64d24ca885ca517aa8ed1be2: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260504 20:02:11.626389 16270 leader_election.cc:290] T 43dc05259beb4569a29aeb5327c9b3bc P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 9 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
W20260504 20:02:11.661774 16427 log.cc:927] Time spent T 83c11e4ac731424b899e837119487d11 P 732c3560a688494196d22a1a48f89968: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260504 20:02:11.640439 16393 consensus_queue.cc:260] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.662962 16393 raft_consensus.cc:3060] T e67d58ddf5464b6daea5f094d46a05b6 P 82787d8e64d24ca885ca517aa8ed1be2 [term 9 FOLLOWER]: Advancing to term 10
I20260504 20:02:11.664206 16300 leader_election.cc:290] T 0f8ca9e82e464c93b50b298fc802e0d0 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), ef6057d9a49e4942b47ea0fcd66ab6f7 (127.11.68.4:41009)
W20260504 20:02:11.659483 16338 consensus_meta.cc:287] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7: Time spent flushing consensus metadata: real 1.486s user 0.015s sys 0.007s
I20260504 20:02:11.666122 16338 raft_consensus.cc:515] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.667295 15605 raft_consensus.cc:2804] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:02:11.668035 15605 raft_consensus.cc:493] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 82787d8e64d24ca885ca517aa8ed1be2)
I20260504 20:02:11.668552 15605 raft_consensus.cc:3060] T c9a8aee69ec8499895f3ab719968d204 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:11.669134 16338 leader_election.cc:290] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [CANDIDATE]: Term 12 election: Requested vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:11.667479 13529 raft_consensus.cc:1240] T 7a80bf1f95a74ce09071a550fe15996f P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 12 FOLLOWER]: Rejecting Update request from peer 82787d8e64d24ca885ca517aa8ed1be2 for earlier term 11. Current term is 12. Ops: []
I20260504 20:02:11.672919 13528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 747) took 1005 ms. Trace:
I20260504 20:02:11.673595 13528 rpcz_store.cc:276] 0504 20:02:10.667029 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:10.677973 (+ 10944us) service_pool.cc:224] Handling call
0504 20:02:10.678550 (+ 577us) raft_consensus.cc:1471] Updating replica for 1 ops
0504 20:02:10.922449 (+243899us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 20:02:10.922499 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0504 20:02:10.923259 (+ 760us) log.cc:844] Serialized 26 byte log entry
0504 20:02:11.423310 (+500051us) raft_consensus.cc:1645] Marking committed up to 9
0504 20:02:11.423386 (+ 76us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:11.423499 (+ 113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0504 20:02:11.672211 (+248712us) raft_consensus.cc:1685] finished
0504 20:02:11.672259 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0504 20:02:11.672534 (+ 275us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":51050,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":270,"thread_start_us":561728,"threads_started":2,"wal-append.queue_time_us":668725}
W20260504 20:02:11.675472 16314 consensus_meta.cc:287] T 86ea905fd808496c9b7d84a57c496cb3 P 5ff0ef72ec3b4667893a0420d3c7f923: Time spent flushing consensus metadata: real 0.768s user 0.013s sys 0.000s
I20260504 20:02:11.676340 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84c093f7c744182b34146f8f1e644d0" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.676766 12865 leader_election.cc:304] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 82787d8e64d24ca885ca517aa8ed1be2, ef6057d9a49e4942b47ea0fcd66ab6f7
I20260504 20:02:11.677505 12820 raft_consensus.cc:2393] T f84c093f7c744182b34146f8f1e644d0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 7: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:11.678485 12865 leader_election.cc:304] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 732c3560a688494196d22a1a48f89968; no voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2
I20260504 20:02:11.638002 16285 raft_consensus.cc:515] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.680562 16223 raft_consensus.cc:2749] T 4472f9104c524735bcfeed8474933a75 P 732c3560a688494196d22a1a48f89968 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 20:02:11.680866 16300 raft_consensus.cc:2749] T c03254a421c44bdf89f865e637b18b2a P 732c3560a688494196d22a1a48f89968 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 20:02:11.682041 12714 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 44807.9us
I20260504 20:02:11.657392 16311 leader_election.cc:290] T bce60695d3db4c06ad8e0eec0419f79f P 732c3560a688494196d22a1a48f89968 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 82787d8e64d24ca885ca517aa8ed1be2 (127.11.68.1:44007), 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415)
I20260504 20:02:11.683323 16285 leader_election.cc:290] T af25f180a18a4e79a93b35bb0a2e55f0 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5ff0ef72ec3b4667893a0420d3c7f923 (127.11.68.2:41415), 732c3560a688494196d22a1a48f89968 (127.11.68.3:39151)
I20260504 20:02:11.678611 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431221deee84429b8261f87972c51218" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.686319 12820 raft_consensus.cc:2393] T 431221deee84429b8261f87972c51218 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 in current term 8: Already voted for candidate 732c3560a688494196d22a1a48f89968 in this term.
I20260504 20:02:11.671027 16362 consensus_queue.cc:260] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 } } peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } }
I20260504 20:02:11.685874 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e67d58ddf5464b6daea5f094d46a05b6" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
W20260504 20:02:11.689250 13497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.68.2:41415, user_credentials={real_user=slave}} blocked reactor thread for 117032us
I20260504 20:02:11.692895 12817 raft_consensus.cc:2468] T f2114545cfec4b19a5a3d1d6751f73da P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef6057d9a49e4942b47ea0fcd66ab6f7 in term 7.
I20260504 20:02:11.687757 12820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fd38813d75c466e8f0502c36726a913" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:11.623626 16278 consensus_queue.cc:1059] T bce60695d3db4c06ad8e0eec0419f79f P 5ff0ef72ec3b4667893a0420d3c7f923 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "82787d8e64d24ca885ca517aa8ed1be2" member_type: VOTER last_known_addr { host: "127.11.68.1" port: 44007 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260504 20:02:11.697129 12818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 2290) took 1033 ms. Trace:
I20260504 20:02:11.696862 12820 raft_consensus.cc:2468] T 8fd38813d75c466e8f0502c36726a913 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 4.
W20260504 20:02:11.701114 12789 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 20:02:11.701006 12818 rpcz_store.cc:276] 0504 20:02:10.657013 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:11.621337 (+964324us) service_pool.cc:224] Handling call
0504 20:02:11.690635 (+ 69298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 20:02:11.702260 12714 leader_election.cc:304] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, 82787d8e64d24ca885ca517aa8ed1be2; no voters:
I20260504 20:02:11.703289 12790 leader_election.cc:304] T 1e115d71520b4beb9b2984dd586354e4 P 5ff0ef72ec3b4667893a0420d3c7f923 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff0ef72ec3b4667893a0420d3c7f923, ef6057d9a49e4942b47ea0fcd66ab6f7; no voters:
I20260504 20:02:11.704145 16215 raft_consensus.cc:2804] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 20:02:11.704643 16215 raft_consensus.cc:493] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 732c3560a688494196d22a1a48f89968)
I20260504 20:02:11.705087 16215 raft_consensus.cc:3060] T 8fd38813d75c466e8f0502c36726a913 P 82787d8e64d24ca885ca517aa8ed1be2 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:11.704944 16314 raft_consensus.cc:493] T 91ff767ed7c24b97940a459421881055 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 20:02:11.710975 13528 raft_consensus.cc:1275] T 91ff767ed7c24b97940a459421881055 P ef6057d9a49e4942b47ea0fcd66ab6f7 [term 11 FOLLOWER]: Refusing update from remote peer 82787d8e64d24ca885ca517aa8ed1be2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260504 20:02:11.711630 12823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53102 (request call id 2239) took 3981 ms (client timeout 2576 ms). Trace:
I20260504 20:02:11.709491 13529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40916 (request call id 742) took 1625 ms. Trace:
W20260504 20:02:11.712055 12823 rpcz_store.cc:269] 0504 20:02:07.730685 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:07.734309 (+ 3624us) service_pool.cc:224] Handling call
0504 20:02:11.711407 (+3977098us) spinlock_profiling.cc:243] Waited 3.01 s on lock 0x7b540014c8d4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:11.711544 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6915651456}
I20260504 20:02:11.712919 16171 consensus_queue.cc:1048] T 91ff767ed7c24b97940a459421881055 P 82787d8e64d24ca885ca517aa8ed1be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260504 20:02:11.713243 12823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57fe021572534bbdbbaea4aa4a64017a" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.713584 16362 raft_consensus.cc:3060] T fea6d3aca49e4b1c9545e2ddb5b8e55b P 82787d8e64d24ca885ca517aa8ed1be2 [term 8 FOLLOWER]: Advancing to term 9
I20260504 20:02:11.714383 12823 raft_consensus.cc:3060] T 57fe021572534bbdbbaea4aa4a64017a P 5ff0ef72ec3b4667893a0420d3c7f923 [term 4 FOLLOWER]: Advancing to term 5
I20260504 20:02:11.715210 13527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40940 (request call id 462) took 1045 ms. Trace:
I20260504 20:02:11.715564 13527 rpcz_store.cc:276] 0504 20:02:10.669389 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:10.675649 (+ 6260us) service_pool.cc:224] Handling call
0504 20:02:10.676290 (+ 641us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 20:02:11.554852 (+878562us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 20:02:11.714955 (+160103us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b540049ded4. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9af1e1 00007f015c9adc1e 00007f0161980ad9 00007f015aeead34 00007f015aeeacc2 00007f015aeeabf1 00007f015aeeab6c 00007f015aee9e22 00007f015ace8bcc 00007f015ace8006
0504 20:02:11.715146 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1288550144}
W20260504 20:02:11.717288 12822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53098 (request call id 2254) took 2594 ms (client timeout 3361 ms). Trace:
W20260504 20:02:11.717832 12822 rpcz_store.cc:269] 0504 20:02:09.123106 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 20:02:09.796052 (+672946us) service_pool.cc:224] Handling call
0504 20:02:11.717011 (+1920959us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b5400374954. stack: 00007f01598dec83 00007f01598deb80 00007f015922c574 00007f0162f076b4 00007f0162f07649 00007f015c917951 00007f015c9b26da 00007f01619821d7 00007f015aeef834 00007f015aeef7c2 00007f015aeef6f1 00007f015aeef66c 00007f015aeee922 00007f015ace8bcc 00007f015ace8006 00007f015ace79af
0504 20:02:11.717204 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3934006528}
I20260504 20:02:11.719131 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ddedcb68670423f9dbf822898141e0f" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:11.719878 12822 raft_consensus.cc:2468] T 7ddedcb68670423f9dbf822898141e0f P 5ff0ef72ec3b4667893a0420d3c7f923 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82787d8e64d24ca885ca517aa8ed1be2 in term 6.
I20260504 20:02:11.720701 12824 raft_consensus.cc:2393] T e67d58ddf5464b6daea5f094d46a05b6 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 732c3560a688494196d22a1a48f89968 in current term 10: Already voted for candidate 5ff0ef72ec3b4667893a0420d3c7f923 in this term.
I20260504 20:02:11.720911 12822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a87aaa2ccb5340acb2654b93a91d22b5" candidate_uuid: "732c3560a688494196d22a1a48f89968" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"
I20260504 20:02:11.721938 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af25f180a18a4e79a93b35bb0a2e55f0" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" is_pre_election: true
I20260504 20:02:11.642347 16303 raft_consensus.cc:515] T fe81ef9fbbe44f89a19e923171496fa5 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ff0ef72ec3b4667893a0420d3c7f923" member_type: VOTER last_known_addr { host: "127.11.68.2" port: 41415 } } peers { permanent_uuid: "732c3560a688494196d22a1a48f89968" member_type: VOTER last_known_addr { host: "127.11.68.3" port: 39151 } } peers { permanent_uuid: "ef6057d9a49e4942b47ea0fcd66ab6f7" member_type: VOTER last_known_addr { host: "127.11.68.4" port: 41009 } attrs { promote: false } }
I20260504 20:02:11.722898 12824 raft_consensus.cc:2410] T af25f180a18a4e79a93b35bb0a2e55f0 P 5ff0ef72ec3b4667893a0420d3c7f923 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82787d8e64d24ca885ca517aa8ed1be2 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260504 20:02:11.723799 12824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e8eba4dbb8d436f81062ca3bbf46788" candidate_uuid: "82787d8e64d24ca885ca517aa8ed1be2" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5ff0ef72ec3b4667893a0420d3c7f923"